►
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Show Me Your Labels and I’ll Tell You Who You Are - Sandor Guba, Cisco
One of the underestimated benefits of Kubernetes is the standardization of labels. Of course in every provision system, you have the ability to mark the assets with taints, tags, labels, or something. Prometheus raised the stakes and built a whole metric system on labels. The concept was so successful that more and more tooling tries to benefit from it. Modern logging, tracing, and metric systems have at least one common characteristic: they have labels. In this talk, Sandor will demonstrate how to fully exploit labels with tools that are available already at your hands. Correlate different inputs, transform logs to metrics, and more!
A
So
I
think
it's
about
time
to
start
hello.
Everyone
thank
you
for
joining
me
today.
I
know
this
is
the
last
day
of
the
conference
and
it's
nearly
lunch
time,
but
I
will
try
to
make
this
half
hour
well
worth
for
you.
So
my
name
is
chandragupta.
I'm
co-founder
and
developer
at
bonsai
cloud
acquired
by
cisco,
I'm
working
for
more
than
five
years
with
kubernetes
and
the
fluent
ecosystem.
A
A
Let's
start
with
the
basics.
Kubernetes
logging
is
not
that
difficult.
The
content
around
time
puts
the
container's
std
out
into
the
host
file
system
and
the
kubelet
service
access
to
these
logs
and
provide
it
to
the
users
through
the
kubernetes
api.
It's
that
simple.
It's
nothing
much!
You
can
configure
about.
A
What's
the
problem
with
this,
the
problem
is
that
these
logs
are
only
stored
on
the
host
file
system
and
eventually
they
will
get
rotated
or
fill
the
node
file
system,
and
the
other
part
is
that
cube
city
logs
is
not
always
the
best
way
to
consume
your
logs.
So
what
you
can
do
you
want
to
collect
these
logs
and
transport
to
your
log
analyzation
system?
A
But
there
are
other
problems
with
this
as
well.
You
need
privileged
access
to
access
the
host
file
system
and
there
is
no
separation
of
logs
on
the
host
file
system.
So
all
kind
of
pods
logs
in
different
name
spaces,
are
just
stored
on
the
same
folder,
but
let's
see
how
this
system
would
look
like
on
kubernetes
and
you
need
the
collector
to
collect
the
log
files
from
the
systems
and
push
them
out
to
an
output.
A
We
collect
all
the
logs
from
the
node
level
and
we
transport
it
to
a
fluid
and
the
stateful
set.
The
heart
of
the
logging
operator
is
a
labor
router
plugin
developed
at
bonsai
cloud,
which
helps
you
to
distinguish
logs
based
on
kubernetes
metadata,
so
you
can
create
different
log
flows
and
if
you
identify
the
low
flow,
it's
really
easy
to,
after
just
apply
any
filter
like
parser,
filter
or
or
or
enhancing
with
guip
data,
and
at
the
end
of
the
log
flow
you
can
post
your
logs
to
any
kind
of
output.
A
That
fluency
supports.
If
we
take
a
look
from
the
other
direction,
how
a
user
would
use
a
logging
operator,
obviously,
as
an
operator,
it
will
use
kubernetes
custom
resources.
There
are
two
main
type
of
these
custom
resources,
flows
and
outputs.
So,
let's
take
a
closer
look
on
these
resources
flow
can
be
a
flow
or
a
cluster
flow.
A
flow
works
within
a
namespace.
A
cluster
flow
works
across
name
spaces.
A
I
don't
have
enough
time
to
dig
deeper
in
the
features
of
logging
operator,
but
since
the
first
release
of
august
in
2018
we
grew
a
firing
community
on
github.
We
have
like
about
thousands
github
stars.
We
have
more
than
half
thousand
people
on
our
slack
channel
and
I'm
really
proud
that
ranchers
official
backend
for
logging
is
actually
logging
operator.
A
A
We
have
a
friend
bob
bob
is
an
infrastructure
engineer
and
he
has
to
operate
an
application
on
kubernetes
cluster
for
the
sake
of
example,
this
application
will
produce
anginix
access
logs
and
he
has
three
requirements.
One
is
that
he
needs
to
extract
metrics
from
those
logs
and
create
an
alert
if
the
500
and
400
error
codes
reach
40
of
the
total
request.
A
A
So
I
will
be
doing
a
live
demo,
so
I
prepared
some
software
on
our
on
my
cluster
and
I
hope
it
will
work
so
on
my
cluster
I
pre-installed
logging
operator.
Obviously
I
installed
promoters
operator
for
matrix
and
I
installed
open
search
for
the
dashboard
and
I
installed
emcom
ui,
which
is
a
cisco
product
on
top
of
logging
operator,
but
we
won't
use
any
of
the
extended
feature,
but
it
has
a
really
nice
web
ui
to
help
you
follow
what
happens
on
the
cluster.
This
is
just
the
default
lucky
flow.
A
So
the
first
requirement
that
we
fulfilled
is
to
actually
install
the
s3
output.
I
won't
show
to
you
guys,
because
this
is
my
aws
credential,
and
I
don't
have
that
credit
card
to
support
you
all
of
you,
but
believe
me,
this
is
the
secrets
inside
this
secret
and
after
that,
I'm
applying
an
s3
output,
and
this
is
how
simple
is
to
just
create
an
s3
output.
You
just
need
to
specify
the
the
bucket
the
region
and
the
secrets
for
aws
account.
A
A
So
so,
if
I
apply
this
on
the
ui,
we
can
see
that
the
custom
resources
I
created
it's
visible
in
inside
the
system
and
eventually
after
reconciling
and
flowing
the
config
reload,
the
log
starts
to
flow
to
the
s3
for
speeding
up
things.
I
will
cheat
a
bit
and
apply
the
last
step
of
the
next
next
example,
because
this
way
we
will
have
some
output
already
during
the
presentation,
but
I
will
show
you
how
you
create
a
new
flow
from
scratch.
A
So
the
first
I
created
a
null
output,
which
is
basically
and
I
applied
again
so
which
is
basically
a
dev
devnuel
device
inside
20.
So
I
will
just
drop
all
the
logs
that
comes
into
fluency,
and
I
need
that
because
I
need
an
output,
but
I
don't
need
the
logs
itself.
I
just
want
to
extract
the
parameters.
Data
from
it,
so,
first
of
all,
my
first
step
is
to
create
a
flow
is
to
have
a
name
of
the
flow.
A
So
I
have
the
raw
logs.
I
pause
them.
The
next
step
is
to
actually
apply
the
kubernetes
filter
on
top
of
it.
So
because
I
have
name
fields
from
the
parts,
I
can
create
a
matrix
filter
with
a
type
counter.
This
will
count
all
the
logs
that
are
flowing
in
and
I
can
apply
labels
like
the
code
label
and
get
the
value
of
the
code
label
from
the
log
itself.
A
So
I
have
matrix
http
response
total
with
each
different
codes
that
are
happening
inside
nginx
access
lab
and
more
over
that
I
will
apply
the
pod
and
namespace
labels
as
well,
because
this
way
I
can
identify
which
ports
actually
produce
the
logs,
because
in
the
parameters
point
of
view
it
will
see
only
the
fluid
pods.
But
with
this
trick
you
can
actually
have
a
source
of
the
pods,
and,
as
I
mentioned,
I
already
applied
this.
So
if
everything
goes
according
to
plan,
I
will
have
matrix.
A
Yes,
I
I'm
gutting
matrix
really
fast
from
from
fluentd,
and
these
are
the
two
hundreds
four
hundred
three
hundreds
extra
and
there
was
one
another
task
that
I
should
create:
an
alert
for
it
and
it's
just
as
easy
as
creating
any
alert
for
for
prometheus,
because
it's
actually
prometheus
matrix.
So
it
may
seem
complex
because
I
have
some
label
magic
here,
but
it's
just
the
simple
prometus
alloy
tool
that
you
would
use
time
I
applied
to
this
cluster
and
I
should
see
in
the
alerts
yeah.
It
was
already
applied.
A
Sorry,
I've
prepared
the
demo
and
it's
a
pending
alert
on
these
logs
that
I'm
outputting
now
so
I
have
one
thing
left
I
need
to
do
the
dji
p
flow.
Actually,
I
can
reuse
the
flow
because
you
remember,
we
didn't
outputted
the
log
anywhere.
One
thing
is
that
you
should
do
before
ingesting
into
elasticsearch.
You
want
to
make
sure
that
the
mapping
is
proper
for
the
data
that
you
want
to
inject,
especially
if
it's
like
data,
because
elasticsearch
need
to
know
what
kind
of
fields
you
are
ingesting.
A
So
I
applied
it
and
let's
see
what
I
configured,
the
first
is
really
simple:
it's
just
an
open
search,
elastic,
open
source.
I
will
use
the
name
interchangeably,
but
in
the
open
search
we
have
the
secret
to
the
credentials
and
we
have
other
locksters
format
and
other
configuration,
and
but
the
interesting
stuff
is
like
we
have
a
did,
a
gip
filter
to
our
previous
form.
It
is
here
and
it
will
automatically
translate
ip
data
to
gip
data
and
the
most
important
part
is
the
location
ri,
which
are
actually
coordinate
pairs.
A
A
Credentials
I
will
log
into
open
search
and
because
I
have
the
mapping
already
in
place,
I
can
to
go
to
the
visualization,
create
a
new
visualization
in
coordinate
map.
I
have
the
nginx
previously
applied
index
mapping
I
can
create
a
bucket
bql
coordinates.
I
can
use
the
gerwish
format
and
the
locational
feed
to
plot
actual
request
original
ap
addresses
and,
as
you
can
see,
it
was
successful
and
we
are
already
counting
the
request
numbers
and
the
original
ips.
A
So
let's
get
back
to
the
presentation
and
with
this
solution
we
actually
fulfilled
all
the
requirements.
We
have
the
s3
bucket
filling
the
logs.
We
have
the
guip
filter
enabled
and
we
have
the
other
things
in
place
and,
moreover,
with
the
kubernetes
labels
attached
to
logs
matrix
and
everything,
it's
really
easy
to
use
attitude
for
correlation,
matrix,
traces,
logs,
etc,
and
are
we
happy
with
that?
Bob
is
not
happy.
Bob
has
a
question
he
what
if
he
wants
to
tail
those
logs?
A
So
if
you
want
to
do
such
things
like
tailing
the
logs,
we
need
a
cli
on
our
computer
that
actually
print
the
logs
on
the
terminal.
We
need
a
service,
we
will
call,
it
lock
socket
that
can
function
as
a
fluency
output
and
we
need
the
way
that
these
two
thing
is
communicating
and
do
we
want
to
add
more
thing
to
do
it,
and
I
would
say
why
not,
let's
make
it
more
interesting
and
add
access
control
to
log
flows
as
well.
A
A
You
just
need
to
send
a
beer
to
come
with
your
requests
and
an
example
of
this
role
would
be
airbag
per
default,
underscore
always
allow
which
means,
if
your
port
has
this
label,
it
will
be
able
alice
will
be
able
to
tell
this
traffic
and
if
it's
denied
it
she's
permitted
to
tell
she
she's
denied
to
tell
this
traffic
and
we
can
define
a
default
policy
like
airbag
per
policy
and
all
lodena.
If
you
want
to
like
you,
know,
have
bots
that
everyone
can
tell.
A
I
know
it
may
sound
a
bit
complex
at
first,
but
let
let's
go
to
some
of
the
architecture,
so
we
have
a
cli
tool.
That's
connect
to
a
lock
socket
after
that.
The
lock
socket
authenticate
against
a
kubernetes
service
account.
It
uses
the
token
and
the
kubernetes
api
to
get
a
username
out
of
it
and
on
the
cli
we
should
have
specify
a
flow
name
and
the
lock
socket
will
create
an
output.
A
A
A
We
got
this
token
out
into
an
environment
variable
for
later
use,
and
we
will
do
the
same
for
both
and
what
I've
done
is
to
update
this
full
service.
The
log
generator
to
have
these
extra
labels,
like
I
described
before
I
have
this
airbox
per
default-
underscore
alice
law
so
always
will
allow
to
trail
the
foo
service,
and
I
have
a
bar
service
which
will
tell
us
that
bob
is
allowed
to
tell
this
logs
and
alice
is
allowed
as
well.
A
Just
we
can
see
it
and
the
last
commands
is
the
kubernetes
state
command,
which
is
our
command,
that
we
just
wrote.
We
dif
we
specify
if
we
want
to
tell
a
flow
or
a
cluster
flow,
we
specify
the
namespace
and
the
name
of
the
flow.
We
give
the
token
and
we
start
to
start
detailing
those
logs
and,
as
you
can
see
in
my
terminal,
it's
already
started
to
tailing
those
logs
and
compared
to
what
I
showed
earlier.
These
are
parse
logs
and
reached
and
in
a
nice
json
format
on
both
sides.
A
So
I
will
upgrade
the
service
to
permit
to
deny
a
list
to
tell
the
bar
service
and
I
will
use
the
same
syntax
like
bob.
So
what
we
have
to
see
now
is
that
actually
each
of
our
users
can
just
access
to
one
service.
There
is
a
sum
of
latency
between
actually
upgrading
the
port
labels
and
getting
the
the
the
per
the
permissions
right,
and
until
that
I
just
wanted
to
show
that
if
we
are
back
on
our
ui,
we
can
see
sorry.
A
Let's
click:
we
can
see
that
what
we
created
before
is
actually
in
place,
so
we
can
see
the
s3
cluster
flow.
We
have
the
gip
flow
down
here
and
you
can
see
that
actually,
the
lock
socket
patched
inside
to
this
gip
flow
and
have
have
this
output
working
for
it
and
another
benefit
of
using
labels
is
actually
you
can
tell
who
has
permission
to
access
support
because
they
are
on
the
pod
labels.
A
So
if
I
just
use
a
a
simple
cube,
ctrl
command
and
check
what
ports
alice
can
access,
I
just
can
print
it
out
so
because
we
are
using
labels,
it's
really
straightforward
to
just
know
what
you
are
doing
in
your
cluster
and
yeah
the
upgrade
finished
and
because
you
know
the
home
chart
reconcile
the
deployment
and
the
pod.
We
will
see
that
both.
A
B
A
B
A
I
don't
really
recommend
to
use
more
than
one
logging.
It
is
possible,
but
it
will
duplicate
the
daemon
set
and
duplicate
the
fluency
as
well.
If
you
need
like
hard
separation,
for
example,
there
was
a
case
when
the
cluster
had
like
hybrid
clusters
with
windows
and
linux
nodes.
So
in
that
way
you
want
to
use
two
different
logging
or
two
different
agent
configuration.
It
depends
on
what
you
want
and
you
can
use
like
namespace,
selectors
or
other
taints
to
not
duplicate
the
daemon
sets,
but
actually
specify
which
nodes
you
want,
which
agents.