►
Description
Cloud Native Logging 101 [B] - Eduardo Silva, Treasure Data
In the Cloud Native Era logging is a fundamental piece of the instrumentation life cycle. With applications running as micro services the log information generated is much more and understanding how to implement and manage logging with this new architecture is fundamental.
This 101 presentation will introduce the concepts of log processing (end-to-end) applied to applications running in orchestrated environments managed by Kubernetes (live demos included).
About Eduardo Silva
Eduardo is an Open Source Engineer at Treasure Data. He currently leads the efforts to make logging more scalable in Containerized and Orchestrated systems such as Kubernetes.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
My
name
is
Eduardo
Silva
I
work
for
this
company,
which
is
called
Trisha
data
and
that
trishal
data
a
we
provide
a
service
in
the
cloud
where
you
can
store
data.
Ok,
I'm
not
going
to
do
a
sales
pitch,
but
it's
really
one
for
the
context.
So
if
you
are
selling
a
service
where
you're
going
to
store
data
in
the
cloud
before
to
store
the
data,
you
need
to
collect
the
data.
A
So
in
order
to
collect
the
data,
we
created
a
project
which
was
called
flu
indeed,
like
five
six
years
ago,
and
nowadays,
flu
Indy
is
part
of
the
CN
CF.
We
always
made
the
flu
in
the
open
source
because
most
of
the
background
of
the
company's
open
source,
so
there
was
no
business
reason
to
keep
it
closed.
So
why
not
make
it
open?
And
what
happens?
A
What
happens
later
is
that
we
had
like
I,
don't
know
many
contributors
use
influen
D
for
their
own
purposes,
and
that's
why
I
asked
Rowan
and
then
we
gave
out
flu
in
d2
to
the
CN
CF
and
continued
to
grow.
So
what
I
do
basically
on
on
treasured
data,
I'm,
a
software
engineer
and
I
work
with
the
fluency
team
and
also
I'm
a
maintainer
of
fluent
bit
project,
which
is
a
prey
that
we
mentioned
this
early
this
morning
and
all
about
a
rate
kubernetes
cloud
native
performance
scalability.
A
So
this
session
we're
going
to
talk
about
login
and
as
I
said
this
morning,
if
we
want
to
come
up
with
a
really
good
solution
for
login,
we
need
to
understand
the
concepts
behind
it.
Okay,
it's
not
that
is
it
as
I
said,
and
going
to
install
just
a
component
a
product,
and
this
will
work
magically
well
at
this
point
is
not
like
that,
because
if
you
have
an
environment
which
is
a
scaling
like
crazy,
you
need
to
scale
login.
A
So
you
need
to
understand
how
things
works
internally,
so
you
can
adapt
your
own
configuration,
improve
your
performance
and
make
it
more
reliable,
so
monitoring
exists
because
we
want
to
monitor
something
or
we
because
we
want
to
troubleshoot
something.
Logging
is
not
just
for
troubleshooting,
because
there
are
many
applications
in
production.
We
generate
login
formation,
but
without
statistics,
meaning
okay.
This
is
the
data
that
my
user
is
using
and
my
mobile
game
application.
A
The
user
from
that
application
that
that
game
is
gener
is
moving
to
this
face
is
purchasing
this
kind
of
information
and
sometimes
in
the
in
the
server
side
they
used
to
login
for
that
purpose.
So
it's
also
monitoring
troubleshooting.
But
if
you
think
about
a
standalone
application,
how
do
they
work?
Basically,
what
they
do
is
to
write
to
a
log
file
or
write
to
a
login
service
like
syslog
or
syslog,
or
write
to
a
remote
service
when
they
chip
the
log
messages
from
point
A
to
point
B,
but
from
a
pressing
operational
perspective.
A
Sometimes
things
can
get
complex,
because
if
you
have
one
application
or
two,
that's
fine,
but
if
you
have
multiple
applications
in
this,
multiple
host
login
becomes
more
complex,
and
when
we
talk
about
the
multiple
host,
multiple
nodes
and
those
kind
of
things,
we
start
talking
about
distributed
systems,
and
that
is
something
very
relevant
for
the
concept
of
kubernetes.
And
how
do
we
design
the
architecture
for
our
applications?
So
if
your
application
is
composed
for
multiple
services,
okay,
sometimes
with
a
couple
application
and
make
different
small
application,
sometimes
this
application
will
run
in
different
nodes.
A
Not
I
mean
a
host,
a
beautiful
machine
and
this
application
and
services
also,
as
I
said
before,
might
scale.
So
from
a
login
perspective,
we
can
have
many
sources
of
information.
Also,
each
application
and
I
am
sure
of
that.
Has
their
own
way
to
do.
Login.
Think
about
that
you're
going
to
have
your
production
application.
You
hire
one
guy
to
develop
one
micro
service
or
their
team.
They
develop
a
different
service
and
what
is
common
is
that
they
are
not
going
to
have
the
same
way
to
do.
Logging
and
I
mean
formats
looking
for
that.
A
Maybe
everybody's
running
to
the
file
system,
which
is
pretty
fine,
it's
pretty
stable
but
having
different
formats,
a
the
pain
and
let
company
do
please,
the
whole
development
teams
are
going
to
use.
The
same
format
is
hard.
People
will
complain,
and
that
does
not
happen.
That
is
true
not
yet,
at
least
so,
if
we
have
also
an
application,
different
applications-
and
we
want
to
do
some
data
analysis.
A
We
want
to
understand
that
some
logs
can
come
from
application,
a
from
host
B
or
from
different
service
or
a
different
cluster,
so
we
need
to
add
the
notion
of
identity
and
that
it's
related
to
metadata.
So
if
we
jump
into
the
cloud
native
space,
this
is
okay.
Cognitive
is
all
about
scaling
resiliency.
But
if
we
talk
about
cloud
native
login,
so
we
say
that
this
is
more
complex
distributed.
Application
is
one
thing
running
a
cluster
of
a
hundred
of
nodes
or
more
even
more
complex.
A
So
we
understand
that
we
have
a
problem.
We
don't
know
yet
how
to
solve
it,
but
a
scale
application
also
requires
requires
two
skill
login.
So
if
we
think
about
how
to
solve
the
problem
in
a
cloud
native
space,
we
need
to
think
about.
Okay.
How
can
it
consume
the
logs
or
the
information
from
the
application
from
different
sources?
Also,
this
information,
besides
the
format,
may
might
have
a
structure
or
may
not
if
it
doesn't
have
it
whoa.
That
means
that
we
need
to
do
some
extra
work.
A
We
need
to
enrich
the
logs
with
metadata
where
these
logs
are
coming
from
and
also
we
need
to
be
able
to
deliver
the
logs
to
a
central
place
for
analysis,
because
log
in
is
not
an
end
right.
Logging
is
just
a
tool
that
maybe
we
can
use
to
take
information
from
one
side
and
centralize
information.
Another
side
to
do
our
own
analysis.
As
I
said
many
times.
Many
people
say
that
log
in
is
boring
and
it
is
you
know
it's
logging,
because
it
happened
behind
the
scenes.
A
It's
not
like
a
dashboard,
you
know,
okay,
you
can,
you
can
do
keep
on
a
staff
la
crystallization,
but
that
is
after
the
storage,
not
in
being
the
main.
While
you
are
collecting
the
data
and
that's
why
logging
becomes
complex
because
you
don't
have
too
much
visibility
from
them
from
that
perspective,
so
who's
familiar
with
a
web
server
logs.
Please
raise
your
hand,
okay,
an
application
like
a
web
server
log.
A
Ok
for
casinos
familiar
if
you
have
a
browser
like
Firefox
Chrome,
that's
the
client
client
site,
but
also
you
have
the
server
side,
which
is
the
web
server,
and
web
server
also
is
registering
to
the
file
system
or
any
kind
of
login
mechanism
who's
accessing
at
what
time
is
accessing
and
what
kind
of
resource
is
trying
to
access
and,
of
course,
may
be
accessing
a
resource
could
be
successful.
Maybe
it
failed,
because
the
result
is
not
there
or
it
does
not
have
the
right
permissions
to
access
to
it.
A
If
we
wanna
do
log
analysis
for
that
special
scenario.
Look
at
this
from
one
special
case:
that's
fine!
But
if
we
have
multiple
applications
in
different
places
that
becomes
hard,
for
example,
in
in
host
a
we
have
three
different
web
services:
okay,
one
of
them
imagine
as
PHP
there
is
Python
and
the
other
may
be
Sebastian,
but
also
have
a
Cosby
which
is
running
my
sequel
with
the
different
formats
for
the
logs.
A
A
So
if
you
were
here
in
the
morning
in
the
keynote
we
described
about
the
login
pipeline,
any
kind
of
problem
that
you
have
even
in
your
life
or
in
software,
a
software
engineering,
you
need
to
come
up
with
a
right
solution.
If
you
understand
the
problem,
you
can
implement
the
solution,
and
that
is
a
software,
that
software
is
a
component
that
try
to
solve
a
problem
but
before
to
write
the
software.
A
A
So
we
need
to
unify
this,
and
that
happens
because
you
can
parse
the
data
once
you
purge
the
data
you
need
to
filter
the
data
and
filtering
can
means
drop
messages,
because
sometimes
you
don't
want
to
have
everything:
a
Huson
Splunk
here,
1
2,
3,
4,
5
yeah,
you
didn't
what
okay
is
as
planned
and
that's
fine,
it's
plug
is
really
good.
The
problem
is
a
commercial
strategy
right,
it's
plug
is
a
data
base,
the
private
data
base
so
and
they
have
the
implement
also
part
of
the
pipeline.
A
A
It
is
really
smart.
You
know
it's
really
bad,
it's
really
good,
but
for
us
it's
not
so
that
specific
use
case
is
skipped.
A
filtering
part,
rental
income
means
drop
messages
or
just
let
pass
these
specific
messages
or
also
obtain
some
metadata
buffering
is
such
as
important
think
about
this.
If
you
are
going
to
chip,
my
looks
to
a
separate
place
which
could
be
a
local
service,
my
network
in
my
cluster,
or
maybe
a
SAS
or
any
kind
of
service
remote
in
the
cloud.
What
would
happens
if
I
have
some
network
outage?
A
I
cannot
chip
the
locks
so
I
had
to
take
some
decisions.
Do
I
want
to
lose
data,
know
some
people
say
yeah,
that's
fine.
I
can
lose
data
because
terabytes
if
I
lost
five
megabytes,
it's
nothing
and
that's
okay,
but
90%
say
no.
We
need
to
have
buffering
why?
Because,
if
I'm
going
to
wrote
the
data
to
multiple
destiny,
one
or
multiple
destinations,
if
something
happens,
I
want
to
be
able
to
recover
from
that
failure
state
and
we
try
again.
The
thing
is
everything
can
fails
and
that
is
fine.
A
Things
fails,
but
you
need
to
be
able
to
recover
from
that
state.
So
the
question
is
what
your
default
behavior
when
things
goes
wrong
and
for
the
login
pipeline,
you
solve
it
with
a
good
buffering,
even
in
even
in
the
file
system
or
in
memory,
and
of
course
that's
our
end
be
able
to
centralize
the
whole
log
in
just
one
place,
and
this
can
be
elasticsearch
plan.
Splunk
can
be
Apache,
Kafka
or
any
kind
of
storage.
A
So
what's
going
it's
a
last
talk.
I
know
that
everybody's
tired,
but
let's
try
it
there's
a
concept
with
that
I
mentioned
before,
which
is
called
the
unstructured
data.
When
people
generate
logs,
usually
don't
have
a
structure.
Okay,
if
you
look
at
that
red
line,
you
will
see
that
Oh
is
a
web
server.
Look
like
I
mean
you
said:
okay,
that
has
a
structure.
Okay
for
you.
It
has
it
because
you
know
that
that
is
an
IP
address.
Do
you
know
that
you
have
a
time
son
when
that
message
was
generated?
A
A
So
if
we
think
about
this,
it
wouldn't
be
better
to
have
a
structure,
because
things
that
your
goal
is
to
do
data
analysis
and
if
you
go
to
do
data
analysis,
you
want
to
say
please
return
me
back
the
whole
records
that
has
a
status
200
and
to
accomplish
that
will
be
better
to
have
a
structure,
and
here
I'm,
not
just
talking
about
JSON
I'm,
just
I'm
talking
about
the
general
instructor,
where
you
can
have
a
map,
a
sense
of
key
values.
So
with
that
information,
any
kind
of
storage
or
storage
engine
can
work
better.
A
And,
of
course,
you
can
optimize
your
computing
time
when
you're
trying
to
search
for
something
so
as
to
total
logs
makes
your
life
easier,
because
you
can
do
also
filtering.
You
can
say:
please
drop
the
whole
messages
and
that
has
a
status
a
500.
We
don't
care
when
the
survey
was
done
for
some
reason
and
also
to
perform
any
kind
of
analytics.
So
from
the
cloud
native
perspective,
we
have
a
login
pipeline,
but
also
we
need
to
be
aware
about
distributed
system.
A
We
need
to
have
strategies
for
applications
and
also
keep
in
mind
that
when
you
want
to
centralize
the
logs
and
that's
what
suin
D
is
flowing,
D
is
an
open
source
project
under
the
Apache
License
and
now
under
the
CN
CF
hands,
which
allows
you
to
implement
a
full
login
pipeline
in
your
architecture,
and
it
does
buffering
buffering
and
all
also
does
filtering,
which
is
really
good.
So,
as
we
said,
we
have
more
than
700
plugins
available.
A
We
have
reliability,
security
and
also
wolf
a
is
written
in
Ruby
and
see
we
have
some
people,
sometimes
that's,
okay.
This
is
written
in
Ruby,
it's
not
too
fast,
but
it
depends
of
a
use
case
and
fill
in
D
is
being
used
for
more
than
2,000
companies.
Just
open
source
version.
Imagine
that
so
Nintendo
is
using
flu
in
D
and
did
you
play
pokemon
go
yeah?
Well,
they
will
think
is
that
that
was
deploy
with
kubernetes
one
of
the
beta
kubernetes
cluster,
and
that
was
running
flu
in
v2.
A
We
didn't
you
enter
enter,
we
read
the
news
was
like
it's
not
like.
Okay,
we
were
working
with
Google
to
come
out.
No,
it
was
not
like
that.
You
know
they
are
pretty
quiet.
So
but
I
said
we
said
also
flowing
D
is
more
than
a
project.
We
consider
flow
in
D
as
a
log
in
full
login
ecosystem,
but
you
get
some
feedback.
You
go
to
conference,
you
go
to
media
apps,
you
talk
to
our
users,
you
go
to
slack
and
sometimes
say
you
know
what
we
need.
A
So
from
that
perspective,
like
two
years
ago,
we
decided
to
start
a
new
project
to
fix
a
some
missing
gaps.
Influen
D
were
six
years
ago.
We
didn't
have
too
much
concepts
about
cloud
native
or
that
kind
of
things.
True,
indeed,
works
really
well,
but
we
also
think
that
we
can
make
it
better.
We
can
make
it
better
as
a
fluent
D,
but
also
as
an
ecosystem,
so
from
a
new
ecosystem.
A
We
come
up
with
fluent
bed
project
and
through
a
bit
with
started
a
almost
two
years
ago,
with
the
same
philosophy,
just
just
try
to
Clement
along
in
pipeline,
but
also
make
it
compatible
will
fluently,
meaning
that
you
can
make
chalk
flu
in
bed
with
fluently
you're,
not
after
India
also
can
talk
over
the
network
with
different
services,
but
also
with
all
fluent
ease,
so
through
embed
also
can
take
out
the
logs
from
your
services
and
talk
to
a
remote
windy
or
any
kind
of
destination.
That
is
supported.
A
A
It
are
very
reliable,
a
solution
for
arm
architecture
and
two
years
ago,
Golan
was
not
really
well
did
not
play
very
well
on
unarmed,
so
and
also
I
think
that
if
we
want
to
build
something
that
really
skills-
and
you
know
you
know
the
language,
because
you
can
make
C
low
walls-
it's
not
just
a
language,
it's
also,
if
you
know
how
how
to
make
it
right,
you
can
increase
performance,
you
can
optimize
from
many
angles.
So
that's
why
we
choose
C.
It
has
a
pluggable
architecture.
We
support
more
than
35
plugins.
A
At
the
moment,
it's
a
synchronous,
I've
been
driven
support,
TLS
security
and
also
monitoring
capabilities,
and
some
of
the
plug-in
that
that
we
have
available.
We
can
tell
log
files,
we
can
listen
from
messages
from
system
D
we
can
parse
is
log.
Messages
is
listen
for
messages
over
the
network
and
we
can
also
get
a
matrix
a
we
were
talking
about
this
today
with
some
guys.
One
of
the
first
plugins
of
fluent
bid
to
get
information
from
was
from
the
CPU
to
measure
the
CPU.
We're
fluent
bait
is
running
as
a
whole.
A
Then
we
added
memory,
disk
network
and
other
MERS
also
as
part
of
the
pipeline.
We
implemented
filters
like
kubernetes
filters.
So
if
you
are
working
in
a
kubernetes
cluster,
as
I
said,
you
want
to
get
some
notion
from
where
the
locks
are
coming
from
or
where
do
they
belong
to?
Because
you
are
not
going
to
query
the
locks
you're
going
to
say.
Please
show
me
the
whole
logs
that
belongs
to
the
pod
name,
X
b-but.
A
You
need
to
have
that
kind
of
meta
data
filters,
allows
you
to
parse
or
to
modify
the
records,
means,
add
your
own
custom
information
and
as
an
output
destination.
We
support
elasticsearch
in
Flook,
CB,
cask
arrest
and
nets
HTTP,
and
we're
going
to
support
calf
canal
native
Kaka
and
the
primary
focus
of
fluent
beat
its
kubernetes
and
toggle.
Of
course,
the
clone
native
space.
It
can
work
everywhere,
but
if
you
said
I'm
using
my
raspberry
pi
and
a
half
a
buck
and
comes
another
person
say
they
have
a
bug
in
kubernetes.
A
Do
you
know
what
we
are
going
to
solve?
First,
so
we
always
say
get
some
pushback
about
that,
but
a
we
try
to
prioritise
I'd
probably
make
some
prioritization
about
the
issues.
So,
let's
jump
into
now
we
talk
about
logging
in
general
right
now.
Let's
talk
about
kubernetes
on
logging
and
for
that
I
need
to
give
you
some
context
very
general
context
about
how
things
works
in
kubernetes.
If
you
have
the
notional
application
that
you
call
of
a
cluster
you're
going
to
deploy
an
application.
A
Basically,
this
application
runs
in
a
in
a
container
right,
but
a
container
in
kubernetes
belongs
to
a
pot
and
a
pod
can
have
many
containers.
So
here
you
start
to
see.
Okay
logging
becomes
complex
because
you
can
have
also
many
pots
inside
a
node
and
not
can
be
a
host
well
made
a
machine
or
a
beautiful
machine
and
in
a
cluster
these
start
to
grow.
Sometimes
you
can
spin
up
more
nodes
drop
those
application.
You
can
scale
applications.
So
how
do
we
implement
a
login
pipeline
at
this
level
to
make
it
right?
A
So
in
kubernetes
there
is
a
concept
which
is
called
demon
set
and
a
demon
set
is
a
pod
that
runs
on
every
node
of
the
cluster
okay.
So
if
you
want
to
solve
login,
the
first
thing
that
you
need
to
do
is,
of
course,
read
the
logs
right,
so
using
flow
embed
of,
as
a
demon
said,
allow
us
with
the
right
configuration
of
course,
to
read
the
applications.
Log
files,
which
belongs
to
the
container
engine
on
this
case
could
be
Duggar.
A
So
once
you
read
the
information,
what
you
do
is
to
go
to
the
API
server,
because
you
want
to
get
some
extra
information
from
that
container.
Remember
that
the
application
does
not
know
that
it's
running
in
a
container,
it
don't
care
about
this
kubernetes.
Oh
it's
running
on
swarm,
it
doesn't
matter,
but
from
a
logging
perspective,
what
we
want
to
solve
me
an
address
is
just
to
give
it
the
notion
about
where
these
logs
belongs
to.
So
what
do
we
do?
A
Is
that
every
time
that
we
are
parsing
the
logs
every
every
log
message
we
go
to
the
API
server.
If
we
don't
have
the
information
and
we
append
the
labels
and
annotation
to
that
specific
record,
we
are
going
to
see
that
now
in
a
demo.
Okay,
so
once
we
collect
the
whole
information,
we
centralize
the
output
in
our
database
or
we
can
send
the
data
also
to
multiple
places.
That
is
one
of
the
flexibility
of
fluency
and
fluent
it.
A
A
Okay,
I'm
running
mini
Q,
which
is
a
single
kubernetes
node
cluster
running
the
computer,
because
I
had
a
problem
this
morning
with
my
real
cluster.
So
we're
going
to
do
is
something
very
basic,
as
you
saw,
we
don't
have
any
pots
running
here,
but
I'm
going
to
deploy
an
application
that
will
generate
just
a
damning
message
to
kubernetes
Adam,
a
message
which
means,
to
the
add,
a
block
message,
so
I'm
going
to
run
it
as
a
deployment.
We
are
going
to
call
it
JSON.
A
A
A
A
Okay,
so
what
I'm
going
to
do
now
is
to
deploy
fluent
bit,
which
will
allow
me,
with
the
default
configuration
that
I
just
ingested,
of
course,
just
to
take
out
the
logs
of
the
whole
pods
or
applications
are
running
and
ingest
that
information
back
into
elastic
search.
So
if
you
want
to
see
the
the
demon
set
file,
it's
a
demon
set
here,
we
have
the
image
right
very
basic.
We
are
talking
to
elastic
search
within
my
local
host,
because
it's
mini
kill,
that's
the
magic
IP,
you
see,
people
address
and
that's
it.
A
A
A
A
A
A
Okay,
here's
some
logs
are
coming
in
okay,
but
what
I
want
to
show
you
is
a
structure
of
the
logs.
Can
you
still
see?
Okay,
so
here
I'm
going
to
do
a
query,
kubernetes
that
and
I'm
going
to
pause?
Okay,
what
I
did
is
I
told
Cabana.
Please
show
me
the
whole
logs
that
comes
from
kubernetes
and
have
some
pot
name
which
start
with
Jason.
Do
you
remember
that
we
started
a
pot
called
Jay,
so
in
the
beginning?
Okay,
that's
it!
A
Okay,
where
they
want
to
tell
you,
is
that
every
single
line
that
was
generated
very
simple
at
the
beginning
in
the
terminal
now
gets
a
bunch
of
information.
That
add
also
adds
a
notion
about
from
where
and
what
this
lock
means.
If
you
look
at
here,
we
have
a
cover,
nearest
metadata
and
check
this.
This
is
the
pot
name,
the
name
space,
the
port
ID
their
labels
and
all
that
information,
and
that
is
really
important.
A
So,
if
you're
going
to
do
login
for
communities
your
login
agent,
it
doesn't
matter
if
useful
in
different
bit
or
not,
but
you
need
to
take
care
about
metadata,
because
that
is
what
you
need
in
order
to
query
the
information
and
if
you
look
carefully,
this
is
the
message
that
I
was
printing
to
the
screen,
but
also
the
logging
tool
was
configured
to
the
composed
that
message
and
make
the
fields
and
expose
them
in
a
different
way.
So,
for
example,
I
will
at
the
record,
which
printed
the
the
number
304,
and
also
has
that
word.
A
A
So
that
was
a
demo
and
what's
next
form
a
fluent
bit
perspective,
we
are
releasing
the
version
0.13
at
the
end
of
the
year.
Well,
actually
December
20
before
Christmas
so
and
it
comes
with
a
huge
news
that
it
comes
with
native
support
for
primitives
for
monitoring.
So
you
can
monitor
the
log
in
Python
or
phloem
bit
where
my
records
are
part,
how
many
bytes
and
consuming
how
much
and
parsing
drop
in
and
send
it
out
and
also
full
integration
with
Apache
Kafka.