►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Getting Started With Logging in Kubernetes - Eduardo Silva, Treasure Data
A good practice when deploying applications in Kubernetes is to set proper instrumentation to gather insights and solve general monitoring needs. Logging is a fundamental piece of the instrumentation cycle and is continually evolving to solve pains associated with unstructured formats, performance and monitoring. In this presentation you will learn the concepts involved in log processing for containerized applications: unstructured/structured logs, log filtering with Lua Scripts, load balancing, scalability and the ability to customize the log processor behavior through declarative resource annotations in Kubernetes within others.
To learn more click here: https://sched.co/FuK2
A
Hello,
everyone,
my
name-
is
Eduardo
Silva
and
today
I'm
here
to
share
our
experience
about
how
to
handle
login
in
kubernetes,
and
everybody
knows
that
login
is
not
the
more
fancy
thing,
meaning
that
login
by
itself
is
boring
right.
You
don't
have
dashboards
cores
or
Phillipa.
Nobody
want
to
do
logging,
but
you
need
it
when
your
application
crashes
or
you
have
some
high
load
or
your
application
is
not
a
responsive
enough.
You
need
to
do
some
kind
of
troubleshooting,
and
that
is
what
it's
logging
forth
and
in
kubernetes
things
get
a
bit
complex.
A
Okay,
so
that
is
this
presentation
about.
My
name
is
Eduardo
Silva
I'm,
a
principal
engineer.
This
company
we
used
to
be
called
Trisha
data.
Now
we
are
part
of
arm
and
a
maintainer
of
a
project
which
is
called
fluent
bed
and
so
in
bed.
Right
now
is
a
break
that
I
just
realized
this
week.
That
is
being
used
a
lot
in
China
by
we
pay
cloud
to
go
and
many
kubernetes
service
providers,
and
we
are
interested
in
logging,
kubernetes,
docker
and
everything.
A
That's
related
with
performance
and
Linux
so
before
to
start
talking
about
a
login
in
Cooper
needed
itself,
we
need
to
start
learning
about
how
applications
works
with
logging
and
the
whole
interactions.
The
most
simple
login
stuff
is
that
when
Apple,
one
application
triggers
a
message
and
that
message
goes
to
a
file
and
that
becomes
logging,
okay
and
if
you
are
familiar
with
UNIX
or
with
Linux
in
general,
you
know
that
every
program
has
at
least
three
stream
interface
right.
A
The
input,
the
standard
output
under
standard
error
and
this
interface
are
really
important
for
the
container
ecosystem.
I
pushed
the
wrong
button
and
that's
alight
so
login
happy
mostly
here
in
the
sandal
output
and
standard
error.
So
that
is
one
way
that
applications
can
start
sending
some
messages
to
the
outside,
saying
something
is
happening
with
my
application
and
maybe
something
is
missing
or
something
is
wrong.
A
Okay,
now
how
this
works
in
log,
in
with
docker
containers
or
with
containers
in
general,
your
application
triggers
a
message
like
hey
some
hi,
but
that
message
is
not
just
the
hi
right.
It
has
a
timestamp
a
time
when
this
message
was
created
and
also
101
stream.
Oh
sorry
about
that,
a
stream
from
where
this
message
was
created
and
well
the
message,
the
stream
and
the
time
itself
and
that
message
is
stored
in
the
file
system
and
the
file
system.
A
I
mean
a
padlock
which
is
bad
leave,
docker
containers
that
the
container
ID
the
hash
and
then
you
get
a
mess,
JSON
map
message
in
the
file
system
and
of
course,
if
you
want
to
do
some
application,
troubleshooting
I'm
sure
that
you
don't
want
to
go
to
the
file
system
and
look
for
that
file,
because
that
is
a
pain.
If
you
have
a
hundred
of
containers,
you
have
a
hundred
of
different
log
files
with
different
hashes,
and
you
don't
know
what
that
means
in
general.
A
Okay.
So
when
working
with
log
in
and
containers,
you
have
to
understand
that
if
one
application
generates
a
message
that
message
go
to
the
file
system
and
then,
if
you're
going
to
do
login,
you
need
to
understand
where
these
locks
are
stored,
how
to
read
them
and
look
up
for
the
specific
fields
inside
the
JSON
map,
which
is
a
lot
here
to
get
the
real
message.
And
of
course
sometimes.
You
would
like
to
obtain
some
metadata
to
that
message.
A
A
A
It
worked
it
for
some
reason,
so
what
we
got
here
is
a
container
ID.
This
is
what
happening
behind
the
scenes.
Actually
you
don't
need
to
care
about
this,
but
it's
good
to
understand
how
it
works,
because
you
can
improve
that
in
the
top
level.
So
we
copying
that
hash.
If
you
go
to
the
file
system,
I'm
going
to
become
sudo,
su
CD
bar
leave,
docker
containers
and
then
I
paste
the
hash
I'm
here.
This
is
one
my
container
environment.
A
There
are
many
files
and
one
of
them
is
the
log
file
I
copy
the
file
cut
there
you
go
what,
if
you
want
to
see
more
pretty
that's
the
message:
okay,
I
took
it
a
message
on
the
command
line
and
that
message
was
stored
in
the
file
system
in
a
JSON
map,
and
then
you
got
it
there.
But
of
course
you
don't
want
to
do
that
manually.
A
A
That
container
is
inside
a
concept
which
is
called
a
pot,
and
a
pot,
as
you
can
see,
can
have
many
containers,
which
means
multiple
applications,
but
in
an
kubernetes
node,
which
can
be
a
very
metal
machine
or
can
be
a
beautiful
machine,
you
have
many
ports
and
a
cluster
is
composed
by
many
nodes,
plus
your
application
server.
We
or
they
master
API
server.
So
if
you
think
about
this,
if
you
are
going
to
do
login
on
this,
you
have
logs
from
this
container
this
order,
which
belongs
to
this
port,
to
this
node
on
this
cluster.
A
So
it's
not
so
simple.
It's
not
just
the
continuity
and
may
look
so
whole
log
in
and
it
contacts
works
in
kubernetes
I
said
we
have
the
application,
we
have
the
containers,
we
have
the
ports
and
the
node,
and
each
one
of
them
has
a
lot
of
data
that
allows
us
to
understand
a
what
is
the
context
of
the
application.
That
is
running
because
for
a
lot
message
you
want
to
understand
what
is
it
from
a
lot?
I
want
to
know
what
is
the
pod
name?
What
is
the
namespace?
What
is
the
host
name?
A
A
It
will
be
a
pain
and
from
a
context,
you
can
say:
okay
from
where
I'm
going
to
get
this
context
for
this
application
that
is
generating
some
log
files,
a
the
first
one
like
a
part
of
the
context
like
a
project,
for
example
the
pod
name,
the
name
space
and
the
container
name
comes
from
the
file
system,
where
it's
running
that
container.
What
is
running
that
application,
but
they
are
more
information
that
is
outside
of
the
context
of
that
node
and
that
information
is
inside
the
kubernetes
master
or
API
server.
A
A
Were
a
notation
of
that
port,
because
with
that
information,
you
can
do
the
fancy
thing,
which
is
called
data
analysis,
because
at
the
end,
log
in
is
just
like
a
tool
to
collect
data
and
aggregate
data
in
just
one
central
place,
because
your
end
is
do
data
analysis,
and
for
that
you
need
to
have
a
log
processor
right
without
the
right
tool
in
you
cannot
do
like
the
old
way
SSH
into
an
OL
and
start
looking
for
the
container
laws.
It
will
not
work
on
that
way.
A
So
when
you
have
the
look-look
processor
working
inside
your
kubernetes
cluster,
then
you
go
to
your
real
end
and
your
real
aim
is
aggregate
and
store
that
information
in
one
central
place,
your
data
base,
a
cloud
service
or
whatever
you're
using
for
your
own
needs.
So
look.
Processing
in
kubernetes
is
like
this.
A
This
is
a
node
with
I'm
running
two
pots
or
two
applications.
Okay
and
these
applications
are
generating
some
login
messages.
Those
login
messages
are
being
trapped
by
the
container
engine
and
being
stored
in
the
file
system,
but
this
is
one
way
to
work.
Some
communities
cluster,
your
system,
D
with
the
journal
service,
but
here
we
are
using
Howard's
with
the
five
system
and
then
what
you
do
is
to
run
a
demon
set
with
a
lock
processor.
Do
you
know
what
a
demon
set
is?
Are
you
familiar?
Okay,
some
of
you,
you
have
a
pot.
A
The
demon
set
is
a
pot
that
runs
on
every
node
of
the
clasp.
That's
it
okay!
So
you
make
sure
that
your
login
processor
will
work
on
every
node
that
belongs
to
one
specific
cluster,
and
then
you
let
your
login
processor
start
consuming
the
log
files
that
are
being
stored
on
that
node.
At
this
part,
we
have
just
one
part
of
the
context
right,
continuty
container
name,
but
then
we
need
to
get
more
information
for
a
for
this.
A
So
the
simple
message
that
we
generated
at
the
beginning
that
actually
was
hey.
Shanghai
and
then
append
the
timestamp
and
the
stream
becomes
something
like
this,
and
with
this
information,
do
you
get
the
real
context,
because
in
your
database,
which
can
be,
for
example,
elasticsearch
or
influx?
Even
you
can
say.
Please
show
me
all
the
logs
that
the
pot
name
belongs
to
what
I
put
up
a
source
ami,
but
it's
coupon
or
which
the
course
belongs,
to
mini
queue,
and
that
is
really
relevant
because
your
application.
A
You
know
that
in
kubernetes
you
deploy
one
application,
but
the
application.
Actually
it's
hard
to
know
at
the
beginning
and
which
node
it
will
be
running.
And
if
you
create
many
replicas
of
that
application,
maybe
will
be
distributed
across
different
nodes.
And
then
you
need
to
centralize
all
that
information
to
be
able
to
query
for
a
specific
patterns
on
the
look
that
was
generated.
A
A
A
A
Watch
typo,
okay,
the
database
is
quite
empty.
What
I'm
doing
here
is
just
go
in
the
database
index
and
try
to
see
how
many
records
do
I
have
in
the
database.
Okay,
okay,
so
getting
back
to
the
kubernetes
cluster.
Can
you
read
the
screen,
then?
Can
you
read
it
right?
Okay,
what
I'm
going
to
do
if
I'm
going
to
the
default
a
namespace
they're,
not
pod
runners,
but
here
I,
have
an
Apache
pot,
like
example,
that
what
it
does
it
just
generates.
A
So
it's
not
a
patch
itself
is
a
support
that
generate
and
simulate
Apache
logs.
So
what
I'm
going
to
do
is
to
run
this
pot
in
my
cluster
and
see
and
see
how
it's
working,
okay,
keep
CTL
logs
Apache
logs,
follow.
Okay,
I,
see
that
my
port
is
generating
a
message
every
second
that
is
totally
random.
Okay,
that's
what
I'm
looking
at
the
logs
from
a
specific
part
of
the
cluster,
but
my
goal
is
ingest
that
log
we're
into
elasticsearch.
A
A
A
Okay,
fluids
running
in
my
note
and
as
I
can
see
now
in
elasticsearch,
the
number
of
documents
is
growing
named
97
98,
99
100.
That
means
that
in
my
local
elasticsearch
I'm
receiving
the
whole
messages
that
are
being
generated
in
the
kubernetes
class,
and
now
we
are
going
to
run
cabaña,
which
is
the
visualization
tool
for
elasticsearch
and
series
even
start.
A
Okay,
let
me
try
this
yeah
that
looks
better
okay,
so
here
we
can
see
that
the
logs
are
coming
to
elasticsearch
in
a
better
way.
Okay,
that
is
the
first
part,
so
we
solve
one
of
the
problems:
how
to
collect
the
looks
from
different
nodes,
different
ports
and
containers
and
aggregating
back
in
one
elastic
search,
and
then
we
can
do
some
kind
of
visualization.
For
example,
let's
look
at
this
record
we're
going
to
look
at
the
JSON
map.
If
you
look
carefully,
you
will
see
that
the
whole
log
is
here.
A
A
Okay,
until
that
it's
fine
okay,
we
understand
how
it
works
and
we're
good.
But
if
you
can
say
this
message,
you
can
understand
it.
If
you
read
Apache
the
Apache
locks,
you
say:
oh,
this
is
an
IP
address.
This
is
a
timestamp,
this
82,
the
tomato,
the
the
address,
the
protocol
version
and,
at
the
end,
the
status
code,
but
for
a
computer
is
not
like
that.
For
a
computer
that
information,
it's
just
a
string
of
bytes,
an
array
of
bytes,
that's
it
is
no
more
than
that.
A
So,
if
you
have,
if
you
want
to
do
some
kind
of
data
analysis
and
process
thousand
of
these
messages,
it
will
take
some
time
so
how
you
can
improve.
That
is
try
to
convert
that
instructor
message
to
a
structure
message,
so
we
are
going
to
a
I'm
going
to
show
you
now
a
different
port
that
we
have
here,
which
is
called
a
Apache
annotated.
A
That
is
a
specific
affluent
bit
filter,
which
is
saying
a
when
deployed,
is
spot
and
flowing
bit,
get
the
pot
locks
and
go
to
the
API
server
and
get
the
annotations.
It
will
find
this
option
and
it
will
say:
oh
this
application
is
using
the
Apache
format,
so
I'm
going
to
apply
my
own
parser
for
Apache
and
then
the
result
will
be
totally
different.
A
B
A
A
The
log
file,
the
log
message-
is
pretty
much
the
same.
Okay,
but
now
we
have
a
new
structure
here.
That's
it's
called
parse,
add
log.
You
can
change
that
in
the
configuration
and
that
change.
If
you
look
carefully
now,
it
says.
Oh,
this
is
the
host
one
for
the
to
something
which
belongs.
To
this
part,
a
user
ID
is
not
set
the
HTTP
method,
which
is
this
one,
the
path
status
code.
A
Okay,
so
hey.
Are
you
familiar
with
fluent
D,
not
fluent
bit
fluency
yeah?
Well,
wendy
is
a
CN
CF
project.
Right
now
we
are
in
the
incubation
phase.
We
are
going.
We
are
working
towards
graduation
and
we
are
just
missing
the
leghul
part,
which
is
you
know,
review
the
trademarks
and
aside
trademarks
to
Linux
Foundation.
But
we
are
working
on
that
and
flew
in
D
is
a
real
login
parent
project.
Flew
in
D
is
running
everywhere,
and
the
good
thing
about
flew
in
D
itself
is
that
it
has
more
than
850
plugins
available.
A
That
is
a
lot
that
does
not
means
that
the
core
team
of
developers
of
Wendy
have
wrote
800
plugins.
That
means
that
the
whole
community
around
flew
Indy
has
done
that
and
in
eruption
a
Google.
You
have
some
of
official
login
agent
for
the
own
platform,
which
is
called
Google
fluently.
If
you
go
to
Microsoft
Azure
services
or
use
OMS,
they
use
the
login
agent
is
fluently
if
you
deploy
the
kubernetes
and
but
in
the
oven
shift
product
from
Red
Hat
the
login
solution.
A
Guess
what
is
fluently
and
we
are
working
towards
evaluation,
but
when
people
was
using
flu
and
D
flu
indeed
was
a
study
like
seven
years
ago.
They
said,
okay,
we're
running
in
kubernetes
now
and
their
specific
use
cases
where
we
need
more
performance
because
they
said
flu
Indy
is
reading
in
a
mix
of
Ruby
and
C
language
and
sometimes
I'm
sending
like
thousand
of
messages
per
second,
and
it
needs
something
faster.
That's
windy
or
something
lightweight
that
do
not
consume
too
much
memory,
but
one
post
because
login
by
itself
is
expensive.
A
It
would
be
always
expensive,
but
you
need
to
find
different
ways
to
how
to
make
it
better
and
faster.
So
three
years
ago
we
envisioned
this
and
we
started
creating
a
new
tool
which
is
part
of
fluenta
project,
which
is
called
flu
embed
and
what
are
the
main
difference
with
between
fluency
and
flu.
Embed
full
and
bid
is
fully
written
in
C
language.
So
if
you
are
a
C
developer,
you
are
welcome
to
join
us
and
try
to
improve
the
project.
A
It's
made
to
consume
a
lot
of
a
low
memory
and
CPU
footprint,
but
in
terms
of
plugins
we
have
just
a
few
45
compared
to
flu
in
D
that
has
more
than
800
right,
because
we
are
most
focusing
on
the
specific
needs.
We
have
built-in
security
with
TLS
and
it's
kuba
is
currently
kubernetes
ready.
So
we
you
have,
you
can
enrich
the
logs.
A
We
I
just
show
you
how
to
ingest
look
from
kubernetes
to
elastic,
search
and
filter
all
the
data
and
come
parse,
the
structured
messages
from
one
place
to
another,
a
what
this
is
like
the
the
white
version
of
the
of
the
pod
definition.
So
if
you
are
deploying
applications
kubernetes
and
you
are
using
fluent
bit-
yes,
you
say
annotation,
because
that
will
save
your
life
or
your
job
life
and
also
flowing
bit.
A
It's
important
monitoring,
because,
if
you're
running
a
cluster,
you
want
to
monitor
everything,
not
just
the
application,
but
also
the
application
that
does
login.
So
we
expose
a
Prometheus
endpoint
where
you
can
see
how
the
data,
how
the
logs
are
flowing
in
that
specific
fluent
bit
node
that
is
running
on
that
kubernetes
cluster
and
also,
if
you
would
like
to
do
some
kind
of
filtering
filtering,
meaning
I
want
to
you
know,
I,
have
this
record
I
would
like
to
remove
some
keys
of
new
ones
or
information.
A
That
is
most
important
for
my
business
like
few
months
ago.
The
only
way
to
do
that
was
write.
A
plug-in
in
C
by
you
know,
write
a
plug-in
for
filter
in
text
time
and
make
it
right
X
more
time
to
so
we
come
up
with
a
solution.
When
now
you
can
run
or
create
your
own
filters
using
the
Lua
language.
Are
you
familiar
with
Lua?
A
A
You
modify
in
a
lower
table
and
then
you
just
return
the
value
that
you
want
and
if
you
use
in
kubernetes,
you
can
add
that
in
a
config
map
which
is
even
better
so
the
good
news
as
a
you
listen
in
this
morning,
we
are
releasing
fluent
bit
one
dot,
o
mq
Khan
in
December
around
December
the
first
week
of
December
I,
see.
There
are
many
features
that
we
will
come
as
part
of
fluent
bit.
That
they
are
not
available,
for
example,
you
get
a
support
format.
A
We
are
coming
also
with
buffering
in
the
file
system,
because
at
the
moment
the
only
buffering
mechanism
of
flow
in
bed
is
in
memory
not
in
the
file
system,
but
we
are
coming
up
with
a
new
design
with
an
iPod
system
and
that
will
be
available
shortly
and
if
you
want
to
contribute
or
join
the
project,
we
have
a
official
documentation
repository.
We
have
a
if
you
are
running
any
Meetup
and
you
want
to
speak
about
login.
Let
us
know
you
can
participate,
of
course
create
more
code.
A
A
There
in
kubernetes,
you
know
we
have
to
understand
he's
asking
a
emotes
of
the
add-ons
for
logging
in
kubernetes
are
with
fluidity,
and
why
not
fluent
bit?
And
when
that
will
happen,
that's
a
question.
The
thing
is
starting
kubernetes,
the
login
is
total
external
login
in
kubernetes
is
not
a
core
component
of
communities.
Kubernetes
just
expose
the
logs
to
the
file
system
and
it's
up
to
you
what
to
use
and
how
to
consume
the
logs.
A
B
A
B
A
A
Well,
do
this
finest
service
most
of
them
are
comments,
went
to
fluish
demo
mode,
not
level,
and
then
you
define
input,
sections,
filter,
section
and
output
sections
and
we
use
the
same
mechanism
between
tax
and
matches
the
difference.
Yeah.
It's
different
this
one
too
big
difference.
We
don't
have
like
two
levels
of
configuration
so
make
it
more
readable
and
the
other
is
that
the
configuration
we
have
like
a
fixed
set
indentation
like
Python
code,
that
you
cannot
break.
If
you
write
wrong
configuration
will
not
work.
Okay,
final.
B
Cut
final
question:
quick
question
is
so
you
tell
ensure
the
annotation
and
as
a
hash,
I'm
touching,
it
will
give
you
very
beautiful
lot
of
Omega.
So
as
we
own
we
are,
we
are
using
the
e
steel
and
is
the
applause.
They
also
provide
a
very
good
longest
and
can
I
get
a
sanitation
fighter
for
the
eto
process.
Yeah.
B
A
A
A
Also,
what
happens
for
log
rotation?
We
solve
it
so
when
I
finally
get
rotated,
because
the
log
file
is
growing
too
much.
What
we
do
is
to
monitor
that
file.
If
a
5
was
rotated
means
that
it
name
has
changed.
So
what
we
do?
We
assume
that
a
file
was
rotated.
We
continue
flew
embed
of
Lundie.
Both
do
the
same
thing,
continue
monitoring
that
file
for
a
couple
of
seconds
and
then
just
stop
reading
on
that,
and
then
we
scan
the
whole
putt
for
new
files.