►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Water, Water, Everywhere: Making Kubernetes Audit Logs Potable - Kate Kuchin, Heptio
Audit logging in Kubernetes is a powerful tool that grants Kubernetes operators more insight into their clusters. Audit logs can tell us what happened in our clusters, when it happened, who did it, what resources were affected, and more. The problem is, even a fairly stagnant Kubernetes cluster generates millions of audit logs per week. And it's up to us to distill value out of what is largely just noise. So, how can we isolate important audit events to better understand what's going on in our clusters? In this session, we'll first go over what Kubernetes audit logs are and what information they provide. We'll then do a live a demo of getting audit set up on a cluster, and inspect the raw logs that are generated. And finally, we'll talk about strategies for pulling useful information out of the deluge, so we can make sense of these millions of audit logs glean actionable insights.
To Learn More: https://sched.co/GrX9
A
Hello,
everyone,
my
name,
is
Kate.
Thank
you
for
coming
today,
I'm
gonna
be
talking
about
audit
logging
and
kubernetes.
Specifically,
the
goal
of
this
talk
is
to
talk
about
what
audit
logging
actually
is.
Go
over
a
couple
of
the
challenges
with
it
and
then
talk
about
how
we
can
address
those
challenges
before
I
get
into
it
quickly.
Introduce
myself
my
name
is
Kate
Cochin
I
am
a
software
engineer
and
help
to
you,
although,
as
you
may
have
heard
literally
a
few
minutes
ago,
hep
geo
has
ceased
to
exist.
So,
as
of
1
p.m.
A
I
am
officially
a
senior
member
of
the
technical
staff
at
VMware.
I'm
super
excited
to
keep
working
on
a
lot
of
the
problems
we've
been
trying
to
solve
a
hefty
at
VMware
and
George
up.
There
he's
excited
about
everything
so
he's
amped
to
add
hep
to
yo
and
now
VMware
I
work
with
a
great
group
of
humans.
A
lot
of
them
were
here
right
now
and
we're
focused
on
building
products
that
make
kubernetes
easier
to
adopt
easier,
to
use
easier
to
bug
easier
to
just
generally
derive
value
from
for
both
cluster
operators
and
app
developers.
A
My
team
consists
of
some
lovely
back-end
focused
engineers.
Those
of
us
focused
on
the
front
end,
a
manager,
Eric
he's
great
and,
of
course,
my
dog
Helen
she's,
the
MVP
of
the
team.
So
what
does
this
group
of
humans
been
working
on
the
lot
of
us
and
working
on
a
project?
That's
designed
to
empower
cluster
operators
to
more
easily
manage
their
multi
cluster
multi-cloud
fleets.
The
motivation
here
is
that
we,
as
we
start
seeing
more
and
more
workloads
migrated
onto
kubernetes
in
the
public
cloud
fleet.
A
The
clusters
are
becoming
more
fragmented,
so
some
people
in
your
company
might
be
running
on
Prem,
some
might
be
using
ATS,
some
might
be
using
GAE
or
Azure
because
they
want
to
use
cosmos
and
data
bricks
and
cluster
operators
need
a
way
to
manage
all
of
these
at
once.
Audit
logging
is
this
super
powerful
tool
that
grants
operators
more
insight
into
their
clusters.
They
can
tell
us
what
happened
in
our
clusters,
who
did
it
when
it
happened
and
what
resources
were
affected?
A
The
problem
is
that
it's
kind
of
hard
to
set
up
an
even
one,
fairly
stagnant
cluster,
generates
literally
millions
of
audit
logs
each
week.
So
if
you're
running
multiple
clusters
across
multiple
clouds,
you
know
it's
certainly
not
unheard
of
for
companies
to
have
hundreds
or
even
thousands
of
clusters.
A
Suddenly,
in
each
of
these
is
spitting
out
a
million
audit
logs
a
day,
it's
gonna
be
pretty
difficult
to
isolate
logs
that
actually
matter.
So
that's
what
we're
trying
to
help
operators
and
app
developers
do
so.
Those
are
some
of
the
things
where
my
team
at
happy
has
been
working
on
just
working
solve
and
we'll
continue
to
be
working
on
the
VMware
I'll
do
a
little
demo
of
what
we've
been
working
on
at
the
end
of
this
talk.
A
I,
don't
think
you
want
to
watch
me
like
use
a
website
for
very
long,
so
it's
gonna
be
quick,
but
if
you're
interested
in
seeing
more
head
over
there
after
you
booth,
we've
got
these
very
fancy.
Ipads
I'm
happy
to
go
more
in-depth,
all
right,
I'm,
a
firm
believer
that
you
should
never
attend
or
schedule
a
meeting
without
an
agenda.
The
same
goes
for
this
talk,
so
briefly,
here's
what
I'll
be
talking
about.
A
First,
we'll
be
going
over
what
kubernetes
audit
logging
actually
is
we'll
talk
about
what
six
questions
and
answers
will
talk
about
Auto
stages
and
we'll
talk
about
audit
levels.
Then
we'll
talk
about
configuring
on
it
and
some
of
the
pain
points
there,
as
well
as
a
new
feature
in
113,
alpha,
dynamic
audit
configuration
which
alleviates
some
of
these
pain
points.
We'll
talk
about
it's
something
my
buddy
and
teammate
Pat
has
been
working
on
upstream
for
a
few
months
and
it's
pretty
cool
I'll.
A
A
So
what
does
kubernetes
audit
logging
I'm
just
gonna,
say
Auto
logging
from
now
on,
because
that's
a
mouthful
and
why
should
you
care
simply
put
audit
logs
report
all
the
requests
made
to
the
kubernetes
api
server
by
admins
users
and
components
of
the
system
itself?
That
means
anytime.
You
apply
some
llamó
to
a
cube,
cuddle,
get
a
cube,
cuddle
watch,
you're,
gonna.
B
A
Some
audit
logs,
but
it
also
means
that
every
time
the
system
does
everything
which
is
spoiler
literally
all
the
time
you're
also
going
to
get
audit
logs.
What
we're
left
with
is
a
chronological
set
of
records
that
document
pretty
much
every
activity.
That's
affected
the
system.
So
why
should
you
care
like?
What
good
is
this
audit
logs
can
be
or
number
things
I
liked
about
them
generally
into
three
categories:
compliance,
anomaly,
detection
and
event:
attribution
a
kind
of
classic
use
of
Auto
logging,
not
just
for
kubernetes,
but
for
all
systems
is
compliance.
A
A
You
probably
know
something
is
going
on
there,
maybe
you're,
seeing
a
lot
of
four
or
three
requests
a
lot
of
403
responses
to
anonymous
requests
like
that
could
be
an
indication
that
there's
bots
crawling
around
looking
for
vulnerabilities.
This
is
a
little
trickier
than
it
sounds
like
we're
talking
about
one
or
a
few
suspicious
logs
amongst
tens
of
thousands
generated
per
hour,
so
most
anomalies
are
not
gonna,
be
immediately
apparent.
A
And,
finally,
the
most
basic
concept
of
audit
logging
is
simply
saying
who
did
what,
when
this
is
sort
of
the
core
value
that
audit
logs
provide,
but
we
could,
but
we
can
expand
on
a
little
further
than
what
you
get
from
just
a
trail
of
audit
logs.
We
can
display
audit
logs
in
a
more
digestible
way.
We
can
filter
out
the
system
events.
We
could
even
start
doing
things
like
event
tracing
where
you
can
follow
a
chain
of
cause,
a
chain
of
causation
and
audit
logs
to
get
to
the
root
cause
of
something.
A
So
if
an
application
goes
offline,
we
could
investigate
audit
logs
and
see
that
the
pod
running
the
app
container
was
still
a
divided,
a
system.
But
if
we
dig
a
little
deeper,
we
can
see
that
someone
either
maliciously
or
accidentally
deleted
the
deployment
that
was
deleting
the
pod
that
was
controlling
the
pod.
So
because
we
have
a
record
of
every
single
event
that
happened
up
until
the
pod
was
deleted.
A
We
can
trace
the
action
through
the
server
and
understand
exactly
what
happened,
who
did
it
when
it
and
and
with
that
data,
we
can
hopefully
understand
a
little
bit
more
about
why
it
happened.
Okay,
that
was
sort
of
a
high-level
overview.
I'm
gonna
go
a
little
bit
deeper
into
what
audit
logs
actually
look
like
so
weak
a
little
deeper
understanding
about
how
we
do
all
of
those
things
with
them.
A
Each
audit
log
should
answer
six
questions.
What
happened
when
did
it
happen?
Who
did
it
from
where
was
it
initiated?
What
did
it
happen
on
and
where
was
it
observed?
This
is
generally
what
we
can
expect
an
audit
log
to
look
like
I.
Don't
expect
you
to
read
it,
but,
depending
on
the
level
and
the
stage,
the
request
is
that
this
blob
is
going
to
be
bigger
or
smaller.
But
all
six
of
these
questions
are
gonna,
be
answered
in
every
log
and
we'll
talk
about
stages
and
levels
after
we
go
through
an
audit
event.
A
So,
let's
break
down
this
event.
So
how
do
we
know
what
happened?
We
have
a
verb
get
if
you're
used
to
interacting
with
kubernetes.
Like
you
cuddle.
These
are
things
you
know
get
watch
patch
create,
that's
usually
what
the
verb
is
gonna
be.
When
did
it
happen?
We
got
a
lot
of
information
about
when
it
happened.
We
had
bunch
of
timestamps
about
the
current
stage
it's
in
when
it
was
created
when
the
request
was
received-
and
we
also
know
this
stage
of
the
audit
log.
Who
did
it
again
a
lot
of
insight
here?
A
We
get
a
user
and
a
username
and
what
groups
they're
a
member
of
so
we
can
see
here
that
this
is
a
system
user.
It's
a
node
and
it's
a
member
of
a
couple
of
groups.
Again
we
can
find
out
a
little
bit
more
about
who
did
it
by
knowing
where
was
initiated,
and
this
is
just
an
array
of
source
IPS
and
what
did
it
happen
on
so
right
now
we
know
we're
looking
at
a
system.
Note
event:
it's
a
get.
A
It
happened
like
three
weeks
ago
whenever
I
made
that
slide,
and
we
now
know
that
it
was
on
a
pod.
So
it's
a
get
of
a
pod
in
the
default
namespace
with
the
name
populate
er5,
whatever
and
whatever.
And
finally,
where
was
it
observed?
This
is
the
same
for
every
single,
auto
log.
It's
literally
always
the
API
server.
A
Every
single
autolog
occurs
on
the
API
server.
If
you
don't
interact
with
the
API
server,
you're,
not
gonna
get
an
auto
log.
So
if
you
are
malicious
user
ssh
into
a
node
and
create
some
binary,
that's
mining,
Bitcoin
or
whatever
other
threat
model
we're
using
these
days.
It's
not
gonna
get
recorded
an
auto
log
because
no
one
has
interacted
with
the
API
server
okay.
So
it's
briefly
what
an
auto
log
is.
Let's
look
at
the
different
stages
of
when
these
logs
are
generated.
A
We
have
a
request
received,
which
is
when
the
audit
handler
receives
the
request
response
started.
We
only
get
those
for
things
like
watch
longer,
running,
requests,
response
complete,
that's
when
the
reserver
is
done
and
no
more
bytes
are
gonna
be
sent.
This
is
aa
blur
we've
dealt
with
it
and
then
panic
occurs
when
a
panic
is
generated
on
the
server.
A
A
We
can
see
what
the
server
response
is,
gonna
be
it
can
either
be
a
no-go
and
the
server
could
panic
and
would
get
a
panic
log
or
if
it's
a
longer
running
request,
like
watch
will
get
a
response
started
and
when
it's
done
response
complete
for
a
lot
of
logs
like
gets,
you
would
likely
see
just
two
of
these.
The
respond
the
requests
received
in
response
complete,
but
the
key
takeaway
here
is
that
for
every
every
event,
you're
not
gonna
get
just
one.
A
The
log
you're
likely
gonna
get
a
couple
of
these
okay,
alright,
so
I
mentioned
Auto
levels
briefly,
I
just
want
to
go
into
what
those
are,
because
audit
levels
basically
could
is
how
we
can
figure
how
much
data
we
want.
Each
audit
log
to
consists
of
so
we
can
look
at
so
we
can
do
none,
which
means
we
don't
want
to
log
anything
about
certain
resources
if
you're
just
not
interested,
you
can
do
metadata.
A
This
is
really
interesting
for
things
like
secrets,
because
you
probably
don't
want
to
be
sending
like
the
full
body
of
secrets
across
the
wire,
because
they
are
a
secret
request.
This
is
the
full
request
body
and
then
the
most
robust
is
request
response.
So
this
is
you
get
both
the
full
request
body
and
the
response
body
response
object
in
your
logging.
So
if
I
were
to
set
a
policy,
it
might
look
something
like
this.
You
can
see.
A
This
is
just
a
kubernetes
object
with
the
kind
policy
I'm
going,
omit
the
stage
request
received
I'm
only
interested
in
metadata
and
I'm
only
interested
in
secrets.
So
what
that
would
come
out
to
is
a
log
that
looks
something
like
this,
and
even
though
this
is
a
fairly
restrictive
policy,
we
can
see
that
it
answers
all
six
of
those
questions
we
talked
about
earlier.
What
happened
if
this
was
a?
We
look
at
the
verb,
it's
a
get.
When
did
it
happen?
You
got
a
bunch
of
timestamps,
you
can
see
all
those
who
did
it.
A
If
you're
interested
in
kind
of
going
into
a
deeper
dive
of
what
I
just
flew
through,
there
is
a
talk
tomorrow
on
auditing,
it's
up
and
on
the
sixth
floor,
it's
at
11:40
in
the
morning.
I
don't
have
time
to
fully
go
into
it,
but
that's
just
a
high
high
level
overview,
and
that
talk
is
by
a
much
hand,
stuffing
all
right.
A
So
now
that
we
hopefully
kind
of
know
what
auditing
is
let's
go
over,
we
know
what
other
thing
is,
and
it
gives
this
so
it
gives
us
insight
into
who
did
what,
when
which
is
super
useful
for
things
like
compliance,
anomaly,
detection
and
event
attribution,
so
we
like
it,
we
want
it.
Let's
do
it
all
right
for
the
bad
news.
First,
you've
got
to
try
to
configure
on
it
before.
A
First,
if
you've
tried
to
configure
out
it
before,
you
know
it's
a
pain
right
and
second,
once
you
have
it
configured
you're
left
with
this
massive
amount
of
law,
because
you
need
to
somehow
make
sense
of
so.
How
do
we
solve
these
problems?
Let's
start
with
configuration
currently,
if
you
want
to
configure
Auto
on
your
cluster,
you
have
to
have
a
really
good
understanding
of
how
your
Cooper
Nettie's
installer
works.
A
You
have
to
know
where
your
kubernetes
installer
looks
for,
manifests
and
then
SSH
into
that
spot
and
copy
up
some
files,
which
brings
us
to
the
second
issue.
Even
if
you
have
a
handle
on
how
your
installer
handles
these
types
of
updates.
Like
you
know,
the
cubelet
looks
for
manifests
and
Etsy
/
kubernetes
/
manifests
on
the
master
node.
You
still
have
to
do
some
pretty
precise
and
often
frustrating
configuration.
You
have
to
set
some
flags,
and
then
you
have
to
copy
up
multiple
files
to
the
master
node,
like
your
Odyssey
pal
audit
policy
file.
A
We
just
talked
about,
and
finally,
once
you've
got
all
that
sorted
out.
You
have
to
restart
the
API
server
to
apply
the
changes.
What's
super
annoying
about.
This
is
even
if
you
made
any
mistakes
which,
if
you're
like
me,
definitely
did
you're
gonna
have
to
go
back
and
do
it
again
and
restart
the
API
server
again.
If
you've
done
this
before
I
would
bet
I
would
guess
you
got
frustrated
otherwise
you're
much
better
than
I
am
at
this
stuff.
Okay!
A
So
what's
the
problem
with
this
beyond
its
annoying,
which
is
the
big
problem,
so
cluster
operators,
you
know
they
have
to
copy
up
files,
the
master,
node
and
then
restart
the
API
server
on
every
single
iteration
of
an
odyssey
pod,
a
policy
file.
So
if
they
want
to
ever
update
their
auto
policy,
they
have
to
SI,
say
SS
ssh
into
the
master
node
figure
out
what
file
they
need
to
update,
hope
they
don't
make
any
indentation
errors,
restart
the
API
server
and
then
do
it
again.
A
If
it's
not
configured
exactly
right,
it's
not
a
trivial
task
and
it's
definitely
harder
than
it
needs
to
be
for
actives,
it's
even
more
frustrating,
because
we're
often
stuck
with
the
configuration
that's
been
set
by
cluster
operators.
So
if
I'm
trying
to
debug
it
debug
an
application
and
I
want
more
than
metadata
figure
out.
What's
going
on
in
the
request
body,
I
can't
turn
up
the
audit
logging
level
without
doing
the
whole
SSH
restart
the
API
server
song-and-dance.
A
A
So
if
it's
not
like
that,
you're
pretty
much
out
of
luck-
and
you
can't
have
like
these
really
cool
third-party
apps,
which
brings
us
to
dynamic
audit
configuration.
This
is
brand-new
in
113
and
it's
still
in
alpha,
but
it
is
super
cool
and
it
alleviates
a
lot
of
the
problems
we
just
talked
about.
So
what
is
it
dynamic?
Audit
config
is
a
set
of
objects,
some
API
server
flags
and
an
audit
sync
up
and
audit
sync
objects
that
allow
you
to
configure
auditing
features
dynamically.
It's
a
good
name.
I'm.
A
Sorry
I
couldn't
come
up
with
a
better
definition,
so
to
enable
audit
on
your
113
cluster.
All
you
need
to
do
is
set
these
three
flags
on
your
API
server.
One
of
them
is
just
turning
on
alpha.
One
is
enabling
the
feature
gate
and
one
is
actually
saying.
Yeah
I
want
audit
dynamic
configuration
when
this
goes
to
GA
you're
only
gonna
need
the
one
flag.
A
This
might
even
be
a
default
flag,
so
you
won't
have
to
worry
about
that.
The
other
thing
you
need
is
an
audit
sync
and
I
would
think,
is
just
a
kubernetes
object
like
a
deployment
like
a
pod.
Like
anything,
it's
just
some
ammo
and
you
can
do
it
with
your
coop
and
you
can
apply
it.
We
think
you
could
all
apply
or
create.
A
So
the
audit
sink
sets
the
logging
policy
and
defines
the
end
point
for
where
you
want
to
send
the
logs
a
couple
of
things
to
note
here
you
have
to
use
a
web
hook
for
dynamic
auditing.
You
can't
just
write
it
to
a
log
file,
so
you
will
need
a
URL
or
a
service
to
send
it.
To.
The
other
thing,
to
note
is
that
stages
are
defined
slightly
differently.
I,
don't
know
if
you
remember,
but
in
the
other
policy
I
showed
you.
A
It
said,
omit
stages
this
week
for
this
new
policy
definition
we
just
say
stages
so
as
a
whitelist
of
stages
you
want
to
receive.
So
how
does
this
help
everyone
out
cluster
operators
once
it's
set
up?
They
can
just
create
audit
syncs
to
their
heart
content,
arts
content
to
set
new
audit
policies.
They
can
update
them.
They
can
create
new
ones.
They
can
enable
third
nuclear
party
applications
super
easily.
A
Actives
can
easily
also
just
create
an
audit
sync
once
it's
turned
on
and
they
can
turn
up
audit
love
audit
logs
so
that
they
can
debug
their
applications
more
easily
and
third-party
app
doves
can
write
an
application
that
includes
an
audit
sync
in
its
definition,
so
they
don't
need
to
worry
about
if
the
cluster
is
configured
properly,
that
they're
dropping
their
application
in
to
okay
super
quickly.
I'm
gonna
do
a
demo
of
it
in
a
second,
but
just
to
make
sure
everyone's
aware
of
a
few
things.
A
This
is
still
an
alpha,
so
maybe
you
don't
go
flipping
it
on
and
prod.
Quite
yet,
there
will
be
an
increase
in
CPU
and
memory.
This
is
because
every
audit
sync
creates
a
buffer
of
5000
events.
It's
fine!
If
you
got
a
few,
it's
pretty
nominal,
but
if
you're
doing
like
a
hundred
syncs,
it's
not
gonna,
be
nominal.
So
don't
do
that.
They
are
working
on
that,
though,
right
now
so
creating
an
auto
sync
means
you
can
pipe
logs
at
any
level
to
anywhere.
A
So
it
should
be
a
cluster
admin
level
coverage
like
you,
don't
want
people
piping
secrets
to
themselves
or
whatever.
If
you
don't
trust
them,
it
doesn't
support
off,
but
they
are
working
on
that.
So
you
will
need
to
get
a
cert
from
the
API
server
and
that's
sort
of
difficult,
so
I'm
not
doing
it
in
this
demo.
It's
a
very
insecure
demo,
don't
do
what
I
am
doing
and
policy
is
scoped
down
for
now,
because
and
so
you'll
need
to
handle.
We
need
to
handle
a
lot
of
like
the
resource
filtering
by
proxy.
A
The
reason
for
that
is
that
they're
introducing
a
new
thing
call
audit
classes,
which
is
gonna,
make
audit
seems
composable.
It's
very
cool,
I'm
happy
to
talk
to
you
about
it
after
the
talk
all
right.
So
let's
do
this
really
quickly,
so
we
can
see
how
easy
it
is.
I
used
kind
for
this
kubernetes
and
docker.
It's
really
easy.
If
you
haven't
tried
it
out,
I
recommend
it
very
outside
of
the
scope
of
this
talk,
but
it's
cool.
A
A
This
isn't
always
necessary,
but
kind
is
currently
at
112
and
I
need
a
113
cluster,
so
I
had
to
build
it
from
source
after
that,
I'm
just
gonna
run
kind,
create
cluster
and
set
this
image
tag
for
this
image
flag
to
use
the
image
I
just
built,
and
then
we
also
have
this
other
flag.
That's
config,
and
this
is
the
kind
config
dot
yamo.
This
is
where
you're
going
to
set
your
API
flag.
So
you
have
audit
enabled
this
is
roughly
what
the
kind
config
yema
would
look
like.
A
You
can
see
here,
there's
some
indented
llamó.
That's
what
that
pipe
thing.
Is
it's
really
ugly,
but
that's
what
it
looks
like
and
underneath
the
pipe?
That's
the
Yambol
in
the
yeah
mole.
That
is
exactly
how
you
would
set
this
up
on
cube
ATM
because
kind
is
running
qadian
under
the
hood,
so
you
can
see.
There's
a
PII
server
object
with
some
extra
args
and
those
three
things
below
extra
args.
Those
are
the
three
flags
we
were
talking
about
earlier.
A
A
All
right,
well
I'll,
just
talk
to
three:
what's
gonna
happen,
so
once
I've
got
my
cute,
so
sorry,
okay,
so
basically
I'm
gonna
run
that
command
I'm
gonna
create
a
kind,
create
cluster
image
and
then
I'm
gonna
set
it
config
and
we'll
see
it
spin
up
and
watch.
You
can
kind
of
see
what's
happening
in
the
screenshot
and
then
it'll
spit
out
a
cute
config
that
I
can
copy
paste
and
then
I
can
interact
with
the
cluster
using
cube
cuddle.
A
It
is
going
now.
Okay,
sorry!
So
now
it's
doing
all
that
stuff
great!
So
we've
got
that
so
once
I
have
my
cute
pickle,
my
cute
config
copy
pasted
I
can
interact
with
this
cluster
now
directly
through
cue
cuddle,
so
I'm
gonna
quickly
build
a
pod.
This
is
the
sketchy
pod.
I
was
telling
you
about
earlier
that
I'm
gonna
spit
like
send
my
logs
to
this
is
not
something
you
would
actually
do.
A
A
Okay,
so
that's
me
getting
my
cute
config
all
right!
So
now
what
I'm
gonna
do
is
I'm
just
getting
the
IP
that
I
need
to
send
the
logs
to
then
I'm
gonna
check
in
my
audits
sink
once
I
have
that
IP
and
make
sure
that
that
IP,
that's
at
my
URL
matches
the
IP
that
the
pot
is
expecting
pretty
simple
and
then
from
here
I'm,
just
gonna
tail,
the
logs
of
that
endpoint
pod
I.
Believe
that's!
What's
happening!
Oh
no!
That
was
me
creating
the
audit
sink
now.
I'm
gonna
tail,
the
logs
voila.
A
We
have
a
bunch
of
audit
logs.
So
this
is
like
what
raw
audit
data
would
look
like
in
your
cluster
there's
a
lot
of
them
and
we
can
see
pre
immediately
that
this
is
like
just
a
lot.
This
is
like
a
small
cluster
on
my
local
machine
that
has
been
out
for
30
seconds
at
this
point
and
we
already
have
a
ton
of
logs
to
go
through,
so
you
can
kind
of
imagine
like
how
out
of-
and
this
would
get
on
a
cluster
of
using
in
production
every
day.
A
Okay,
so
there's
some
great
work
going
up
going
on
upstream
to
help
alleviate
the
pain
of
configuring
on
it.
But
how
can
we
make
sense
of
the
audit
data
once
we
have
it
set
up.
I've
mentioned
this
a
few
times,
but
I
just
really
want
to
emphasize
like
how
much
data
we're
talking
about
it's
a
lot.
It's
like
a
preposterous
amount
of
data
as
an
anecdote
when
we
first
started
building
this
building
out
this
feature,
I'm
gonna
show
off
in
a
minute
ross
or
designer
design.
A
This
view
with
something
like
142
logs
per
hour
and
we're
like
oh
yeah,
cool
that
looks
good
great
graph.
Uh-Huh
seems
reasonable,
but
once
we
hooked
it
up
to
a
real
cluster
and
realized
like
just
like
how
aggressively
wrong
we
all
were.
We
were
seeing
something
like
a
million
events
every
day
and
we
had
to
pretty
seriously
rethink
how
we
wanted
to
display
that
data
so
I,
just
just
to
give
that
some
color
I
wanted
to
give
an
example
of
one
interaction
with
the
API
server.
A
So
if
I'm,
a
developer
and
I
just
want
to
create
a
simple
deployment,
assuming
this
command
successfully
creates,
you
can
expect
to
get
at
least
three
logs
that
are
attributed
to
me
again.
That's
checking
to
make
sure
the
deployment
I'm
trying
to
create
doesn't
already
exist.
That's
gonna
fail,
I'll,
create
the
actual
creation
of
the
planet
and
then
a
list
just
making
sure
the
deployment
is
there
and
everything's
copasetic.
So
three
events
for
that
one
action.
Meanwhile,
the
system
generates
around
40
events.
A
For
that
one
action
kubernetes
has
been
up
a
replica
set,
the
pods
do
gets
updates.
I
know,
there's
like
a
bunch
of
gets
that
happen
at
the
end.
It's
a
lot
and
I
couldn't
even
fit
all
40
on
one
slide,
but
these
do
actually
represent
like
a
real
engine.
X
deployment
that
I
spun
up
to
give
it
a
little
more
color
I've
got
a
pretty
stagnant
cluster
I
keep
up
that
has
awed
it
enabled
on
it.
A
I
really
don't
interact
with
this
cluster
at
all,
so
the
logs
it
generates
are
just
logs
generated
by
the
system
to
keep
the
cluster
healthy.
So
this
is
the
cluster
that
pretty
much
like
exists
in
my
heart
and
on
my
AWS
bill
and
that's
it
and
it
is
still
spitting
out
60,000
out
of
events
per
hour,
so
that
deployment
that
generates
40
events.
A
That
seemed
like
a
lot
in
comparison
to
the
three,
but
when
your
wait
waiting
through
60,000
events
in
an
hour
like
isolating
those
40
events,
much
less
than
three
events
becomes
pretty
difficult
unless
you
know
exactly
what
you're
for
which
brings
us
to
what
we've
been
working
on
and
have
to
you.
So
our
team
has
been
working
on
a
product
if
it's
aiming
to
make
audit
logs
something
more
than
just
a
lot
of
data
about
your
cluster,
we
think
there's
a
ton
of
value
that
can
be
derived
from
audit.
A
If
you
just
know
to
where
to
look
so
our
goal
is
to
start
surfacing
information
to
cluster
operators
that
is
about
it
readily
available
in
audit
logs,
but
isn't
easily
findable.
Unless
you
know
you're
looking
for
in
a
lot
of
ways
here,
audit
is
just
a
piece
of
the
puzzle.
If
we
don't
know
something
is
wrong
or
we
don't
know
what
we're
looking
for
audit
logs
aren't
going
to
alert
you
to
the
fact
that
something's
amiss
in
your
cluster.
This
isn't
like
an
extra
layer
of
security,
but
what
you
do
know
something
is
wrong.
A
We
can
use
audit
to
figure
out
why
it's
wrong.
Okay,
I'm
gonna
have
to
hope
this
works
all
right.
So
this
is
the
project
that
we
have
been
working
on.
You
can
see
here.
I've
got
a
fleet
of
clusters
like
I,
should
see.
We've
got
some
gke
clusters.
We
got
some
PKS
clusters,
they
have
this
AWS
cluster.
That
has
a
warning
that
there's
a
disk
pressure,
so
I
click
into
that
and
then
I
would
see
this
cluster
overview.
Page
or
I
can
see.
A
Yep
I've
got
two
notes,
and
one
of
them
appears
to
have
some
kind
of
warning
at
that
point.
I
could
investigate
a
little
bit
further
go
look
at
some
nodes
and
yep
that
doesn't
look
great,
just
pressure
a
little
bit
higher
than
I'd
expect
for
the
allocated
CPU.
So
at
this
point
once
I
know,
there's
something
up:
I
could
go
into
audit
and
try
to
figure
out
why
it's
up.
A
So
they're
taking
my
time
on
this
one,
all
right,
okay,
so
this
is
the
audit
page
and
right
off
the
bat.
We
can
see
that
there
are
37,000
events
in
the
last
hour
and
that
looks
like
even
with
some
sort
of
dip
in
the
API
server
there
might
have
been
an
outage,
which
is
why
it's
a
little
lower
than
60,
but
it's
still
pretty
consistent
like
there's
not
really
a
lot.
You
can
tell
there's
like
a
little
bit
of
a
spike
here
and
there,
but
nothing
that
would
necessarily
bring
off
any
alarm
bells.
A
This
is
a
little
more
useful
right,
like
suddenly
we're
seeing
a
couple
of
spikes
and
like
that,
could
very
well
be
the
problem.
That's
going
on
the
spikes
before
are
me
messing
up
the
demo
the
first
time,
but
those
last
spike
is
the
one
that
will
go
in
and
look
at.
So
if
we
scroll
down,
we
can
see
this
overview
of
all
of
the
auto
logs.
So
this
is
the
same
exact
data
I
just
showed
there
was
like
blobs
of
JSON
on
determine,
but
it's
just
displayed
a
little
more
nicely.
A
So
we
can
see,
we've
got
some
very
bad
scary,
app
stuff
going
on
that.
Doesn't
sound
good,
so
I'm
gonna
click
into
that
and
see.
If
I
can
answer
those
six
questions
we
talked
about
earlier.
You
know,
I
can
see
here
that
it's
a
create
I
can
see
who
created
it.
Fabio,
typical,
Fabio
and
I
can
see
what
it
what
namespace
it's
in
so
I'm.
You
know
where
to
go.
Look
for
it.
I
can
see
the
source
IPO
is
created
for
I
can
figure
out.
A
All
of
these
things
I
need
to
do
I
need
to
know
as
a
cluster
operator
to
go
and
actually
fix
it.
So
at
that
point
you
know
this
is
I
could
go
into
my
terminal
and
just
really
quickly
do
a
get
of
my
pods,
the
namespace.
A
lot
of
typos
in
this
I
don't
feel
like
we're
doing
it
there.
They
are
very
bad
scary,
app
52
seconds
old
and
I'm.
A
Just
gonna
delete
that
pod
because
I'm
pretty
sure
it's
suspicious
and
so
once
I
deleted
it
I
could
go
back
to
HQ,
go
back
to
the
project
and
see
that
the
cluster
is
healthy.
Again,
there
was
a
little
green
dot
next
to
the
cluster
name
and
I
can
go
in,
and
I
can
filter
by
very
bad
scary
app
and
see
not
just
the
events
that
Fabio
created
and
the
system
created,
but
also
the
events
that
I
created
when
I
went
and
deleted
the
deleted
that
pod.
A
Okay,
I'm
about
done
so
when
I
was
preparing
for
this
talk.
I
had
people
ask
me
what
I'd
be
talking
about
when
I
reply
with
audit
logging,
yeah
I
pretty
universally
be
met
with
either
it's
very
blank
stares
or
something
along
the
lines
that
oh,
that
sounds
like
really
boring.
I
think
both
of
these
reactions
kind
of
capture,
the
two
problems
we
have
with
audit
right.
The
blank
stare
comes
from:
what
on
earth
is
audit
logging
drive
and
have
that
do
I
want
that?
A
A
It,
set
up
an
understanding
why
you
might
actually
want
to
look
at
that
pile
of
juice
and
one
of
our
goals
for
the
next
few
months,
both
upstream
and
and
the
things
we're
gonna
be
building
it
VMware
is
to
keep
addressing
these
problems
when
dynamic
thought
it
gets
to
GA,
for
instance,
and
maybe
it
will
maybe
even
be
enabled
by
default.
It
really
will
just
be
as
simple
as
creating
audit
sinks
to
get
auto
policies
set
up
and
once
they're
flowing.
A
We
hope
to
keep
building
out
solutions
like
what
I
just
showed
you
to
make
them
useful
and
hopefully
a
little
bit
less
boring.
So
with
that,
thank
you
all
for
coming.
I
will
be
over
the
fgo
booth
for
a
few
hours
or,
if
you
prefer,
you
can
find
me
on
Twitter,
where
my
DMS
are
not
and
never
will
be
open.
But
you
can
at
me
thanks
again.