►
Description
Audit in Kubernetes Now, and in the Future [B] - Maciej Szulik, Red Hat
Quoting Wikipedia “an audit is a systematic and independent examination of (...) records”. Now think for a second, how much information is floating through your Kubernetes cluster. Deployments, Jobs and many other controllers creating and destroying Pods. Administrators creating Users, granting Roles. Users creating and modifying ConfigMaps, Secrets and many, many others. You can limit actions performed by a single User creating Roles, controllers can be assigned ServiceAccounts, etc, of course. But even with all that in place, are you sure you can easily trace when a change was introduced, and most importantly who performed it? This is when auditing comes into play.
During this presentation, I will introduce what auditing is, and what you can expect from one of the best hidden features of Kubernetes, and why should you care. I don't like just talking about ideas, so we’ll also walk through a live demo showcasing the audit feature. With all the current state laid out, I will discuss the future evolution of this feature. Most importantly, I will cover the scope of the information that should be gathered during processing each request. What policies should be implemented to provide reasonable balance between performance and accountability. Lastly, I will cover the most sensitive topic, how to store all that information.
After this session you will understand how auditing in Kubernetes works, and how to leverage it to stay informed about what goes on in your cluster. Furthermore, I am hoping this presentation will foster a discussion about advanced audit feature and its shape in Kubernetes.
About Maciej Szulik
Maciej is a passionate developer with over 10 years of experience in many languages. Currently, he is working on OpenShift and Kubernetes for Red Hat, whereas at night he is hacking on bugs.python.org and CPython's IMAP library. In his other spare time he organizes PyCon PL, talks at various events and meet ups around Europe.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hi,
oh
I
do
hope
that
the
broken
glass
at
the
beginning
of
my
presentation
will
bring
me
luck,
but
just
in
case
be
warned
if
I
start
dancing.
That
means
the
the
floor
is
wet,
so
it
might
happen
to
me.
So
let
me
start
first
off
with
we
are
are
humans
and
we
do
mistakes
if
you're,
not
a
human,
and
if
you
do
not
mistakes,
I
think
you
should
better
leave
this
presentation.
That's
that's
how
it
works.
So
my
name
is
machi
and
I
work
for
Red
Hat.
A
You
can
find
me
pretty
much
everywhere
in
the
internet
under
Soltis.
That
includes
github
twitter,
even
at
the
Red
Hat
who
have
you
were
recently
touched
or
had
a
connection
or
was
disrupted
by
the
s3
problem
about
a
couple
weeks
ago.
I
do
believe
some
of
you.
Okay,
so
have
you
guys
get
a
chance
to
read
the
post-mortem
of
that
event?
Let
me
quote
you
what
what
they
wrote
so
an
authorized
s3
member
using
established
playbook,
executed
a
comment
which
was
intended
to
remove
a
small
number
of
servers
for
one
of
the
s3
subsystems.
A
Unfortunately,
one
of
the
inputs
of
the
command
was
entered
incorrectly
and
the
larger
set
of
servers
was
removed
that
intended.
Okay.
That
makes
sense.
But
the
question
is:
are
we
able
to
know
what
is
going
on
in
the
kubernetes
cluster?
We
are
serving
billions
of
requests
every
hour,
depending
on
the
traffic
that
you
have.
Are
you
able
to
track
what
happened
or,
most
importantly,
who's
responsible
for
the
problem?
What
was
the
problem?
Was
it
a
malicious
user?
A
Was
it
a
mistake
by
a
user
I'm
going
to
talk
about
the
auditing
future
that
we
currently
have
and
the
kubernetes
and
hopefully
what
the
future
holds
for
for
that
feature?
So
let
me
tell
you
a
story
how
the
current
editing
future
that
we
have
was
brought
to
life.
So
sometime
last
year,
Mike
came
over
to
me
and
poked
me
and
he
said
ma
hey.
We
need
very
simple
logging
feature.
A
You
know
kind
of
like
we'll
say
that
we'll
say:
oh,
this
user
access
that
URL
and
that's
it,
and
it
would
be
nice
if
it
if
it
also
logged
the
returned
code,
so
I
started
poking
with
David
eats
and
Stephane,
and
we
figure
out
that
it
will
be
the
best
option
currently
to
create
a
additional
filter.
So
every
single
request
that
comes
into
API,
cube,
API
server,
goes
through
a
bunch
of
different
filters.
A
Currently,
that
includes
request
contacts
which
creates
a
context
for
the
entire
request,
and
it's
then
reused
or
eventually
filled
in
with
additional
metadata.
That
includes
request
info
we're
checking
how
many,
how
many
requests
are
in-flight
we
check
if
none
of
the
requests
is
is
not
exceeding
the
timeouts.
A
A
You
can
basically
turn
on
the
the
outlet
and
currently
with
passing
some
of
the
the
flags
the
most
important
one
which
actually
enables
the
outer
tank
is
the
last
one
added
lock
path
that
can
be
either
a
a
file
where
the
log
file
where
the
auditing
will
be
written
to
or
a
if
you
want
to
get
that
STD
out
the
other.
Allow
you
to
roll
the
file
when
it
will
be
change,
how
many
entries
in
and
how
about
the
size,
etc.
The
the
usual
rolling
of
the
files
so.
A
A
Eventually,
if
we
have
that
information
about
groups,
we
also
keep
the
information
if
the
user
was
a
is
working
as
himself
or
he
was
impersonating
and
in
the
third
case
where
I
was
working
as
a
user.
Well,
I
was
from
a
user
token,
but
I
was
impersonating
as
an
admin.
In
the
last
case,
I
have
a
clear
information
that
I
was.
A
Although
I
am
a
user,
the
ass
is
filled
with
with
an
admin
that
I
was
working
as
the
two
requests,
because
the
time
when
the
request
comes
and
goes
out
are
correlated
by
the
ID,
that
is
uniquely,
that
is
generated
upon
the
initial
request
and
then
so
that
you
can
actually
correlate
the
response.
That's
that's!
Basically,
that's
basically
it
about
the
demo,
so
very
simplistic,
as
I
said,
but
at
a
time
when
I
was
writing.
Writing
the
outed
back
then
we
didn't
have
the
feather
the
aggregated
cube
server.
A
Audit
does
not
provide
additional
security
to
your
system,
there's
some
people
who
would
like
to
have
it
that
way.
I
was
recently
struggling
with
an
idea
that
somebody
would
like
to
be
able
to
figure
out
if
his
server
is
ddosed
by
looking
at
the
outer
lock,
but
I
said
like.
Currently,
we
do
not
do
that
because
the
audit
is
after
authentication,
so
only
outside
attenti
kated
users
are
actually
locked.
Oh
one
thing
that
I
didn't
mention
a
minute
ago
is
that
the
outer
locks
currently
also
works
for
the
non
secured
endpoint.
A
So
if
you're
hitting
the
not
secure
err,
point
you'll
just
have
the
information
about
users
groups
as
user
as
group
those
will
be
empty.
So
obviously
we
want
to
work
with
secured
cube
system.
So
let's
get
go
back
to
some
boring
stuff
and
the
definition
of
the
auditing.
Well,
if
you,
if
you
look
internet
about
the
auditing,
the
majority
information
that
is
there
is
about
performing
auditing
of
the
systems,
but
not
a
lot
about
how
to
write
a
system
for
performing
some
kind
of
a
log
trail
what's
happening
in
the
system,
etc.
A
There
is
a
missed
definition,
which
clearly
states
that
and
out
it
is
maintaining
a
record
of
activity
of
users
or
other
components
of
the
system,
and
that's
that's
that's
very
important,
because
currently
our
audit
treats
evenly
all
the
components
of
the
system
that
includes
users
but
as
well
controllers
nodes
and
everything.
Every
single
element
of
the
entire
cube
that
is
reaching
the
API
server
will
be
logged
I
explicitly,
if
you
remember
correctly,
I
explicitly
grabbed
through
the
audit
logs
just
to
get
the
information
about
the
pot
requests
that
I
was.
A
If
I
did
not,
we
would
see
tons
of
tons
of
get
requests
for
for
jobs
for
for
deployment,
for
replicas
set
for
parts,
etc,
etc.
From
all
of
the
content,
from
all
of
the
other
controllers
and
components
of
the
system
and
as
I
said,
audit
trail
can
assist
in
detecting
security
violation,
but
it
is
not
a
replacement
for
any
of
those
systems.
I
did
some
more
googling
and
searching
for
resources
where
we
could
base
our
current
work.
Around
auditing
I
used
to
I
used
to
write
one
system
for
auditing.
That
was
just
well.
A
We
want
to
do
this
and
that,
and
we
would
like
you
know
that
fits
in
what
the
customer
wants.
We,
we
just
followed
that,
but
this
time
I
wanted
to
do
it
properly.
So
we
did
with
Stefan
a
little
bit
of
Investigation
and
actually
OpenStack
is
using
cloud
cloud
auditing
data
Federation
for
defining
how
the
audit
should
look
like
and
actually
the
next
information
has.
A
Some
only
has
some
short
description
and
definition,
but
the
CA
DF
has
a
really
nice
definition
of
what
should
be
part
of
every
single
audit
log
and
it
is
framed,
and
it
is
framed
as
7.w
questions
so
when
when
did
it
happen,
what
happened,
who
initiated
it
on?
What
did
it
happen
from
where
and
to
where
and
where
was
it
observed?
Now,
let
me
map
you
that
questions
to
the
current
outer
log
that
we
have
what
happened.
A
We
obviously
log
the
HTTP
method
that
happened,
that
it
will
be
get
delete,
passed
put
what
else
we
do
watch
list.
We
do
that
when
did
it
happen?
Well,
obviously,
there
is
a
timeout
there's,
an
information
about
when
the
request
came
in
and
when
their
response
was
was
sent
out
and
who
initiated
it.
There
are
four
and
four
field
of
information
who
in
who
actually
is
behind
every
single
request.
A
Obviously,
when
we
will
be
using
the
the
non-secured
endpoint
that
will
be
empty,
which
is
in
majority
of
cases
pointless
so
but
we're
let's
focus
on
the
secured
endpoint,
we
know
the
user.
We
know
the
groups
that
he's
part
of
we
eventually
know
if
he
was
working
as
himself
or
he
was
impersonating
on
what
did
it
happen?
So
here
we
just
present
the
URI
of
the
end
of
the
requests
that
this
request
was
targeting
and
from
where
was
it
in
initiated?
Obviously
that
will
be
the
other
clients
IP.
A
No,
so
there
are
two
unanswered
questions,
although
one
of
them
has
a
really
simple
answer:
where
was
it
observed?
I
told
you,
I
showed
you
at
that
third
slide.
It
was
observed
at
the
API
server
level
and
that's
fine
when
we
are
having
just
one
API
server,
but
unfortunately,
with
the
aggregated
cube,
we
can
have
multiple
API
servers.
Even
more
and
I
was
talking
with
Niall
Niall
yesterday
we
can
have
a
federated
cube
server
and
in
those
cases
you
want
to
know
the
level
at
which
particularly
Quest
was
locked.
A
You
want
to
know
whether
this
happened
on
API
server,
a
B,
see,
or
else
you
want
to
know
if
it
if
it
went
through
a
cube
aggregator,
whether
it
originated
from
from
federated
server
or
whether
it
originated
from
a
and
directly
from
me
from
a
client.
So,
as
I
said,
the
current
solutions
has
its
pros
and
cons.
Obviously,
one
of
the
big
prize-
it's
very
lightweight,
it's
almost
negligible
via
hat
of
of
the
current
implementation
and
it's
very
simple
format.
The
format
is
Apache
access.
A
Log
similar
like
so
it's
very
easy
to
parse
that
or
use
existing
tools
to
parse
the
information
from
from
the
thing,
but
unfortunately
it
is
only
HTTP,
so
there
are
no
deeper
enter
in
inspection
into
what
was
modified.
How
the
actual
object
that
was
passed
through
looked
like
what
was
happening
with
the
object
and,
as
I
said,
it's
very
noisy.
A
Currently,
the
log
is
tons
of,
has
tons
of
tons
of
information,
and
majority
of
it
is
coming
from
other
components
of
the
system
which
is
in
some
cases,
is
perfectly
okay,
because
you're
you
want
to
have
everything
log,
but
there
are
cases
that
you
want
to
focus
only
on
certain
area
and
and
what
I
said
as
as
well
at
the
beginning,
it
is
currently
only
log
file
biased.
There
is
no
possibility
to
send
the
logs
to
a
central
file.
So
what
is
the
future?
A
The
feature
is
the
feature
22.
This
is
where
the
auditing
discussion
is
happening.
This
is
where
it
everything
that
I'm
talking
about
originates
from
and,
most
importantly,
the
community
pool
number
145.
This
is
where
Stefan
and
I
created
a
proposal.
How
the
advanced
editing
that
I'll
be
talking
in
a
minute
about,
should
look
like
what
should
be
their
features
when
we
what
we
should
be
able
to
do
and
what's
possible.
A
So
let
me
quickly
go
through
the
proposal
and
and
let
you
off
to
enjoy
the
afternoon
party
so
the
biggest
question
that
we
we
wanted
to
answer
and
although
we
had
the
implementation
currently
in
place
inside
the
API
server,
we
wanted
to
rethink
that
idea
and
ask
ourselves:
do
we
really
want
to
bake
the
auditing
inside
of
the
API
server,
or
maybe
it
will
be
doable
to
have
an
external
solution
outside
of
the
API
server?
But
after
some
thinking
we
figure
out
that?
A
Well,
the
external
approach
doesn't
have
that
good
knowledge
of.
What's
going
on
in
cube,
it
doesn't
have
the
necessary
information
about
the
the
objects
that
are
being
processed
by
the
API
server,
whereas
inside
of
the
API
server,
we
can
clearly
go
all
the
way
down
to
the
storage
level.
So
that's
why
that's
why
the
the
later
approach
one
and
there
are
four
main
concepts
that
we
want
to
introduce
with
the
new
proposal?
Most
importantly
event,
the
outer
event
is
the
actual
structure
that
will
hold
the
entire
information
about
every
single
request.
A
That
includes
and
I
need,
to
figure
out
to
my
my
notes.
That
will
include,
obviously,
that
information
that
we
currently
have,
which
is
time,
stop
source
IP
method,
your
user
information
namespace.
But,
additionally,
we
would
like
to
decode
the
group
version
kind
information.
The
response
code
like
previously
and,
most
importantly,
request
objects.
Maybe
at
some
point
in
time
we
will
be
able
to
provide
you
with
the
diff
information.
A
What
has
changed
and
the
in
the
object,
the
policy,
the
policy
and,
let
me
go
back
to
the
event
once
again
and
the
event
object
will
be
filled
by
all
those
different
layers
and
then
pushed
over
to
to
the
output
backend
the
policy
which
will
describe
which
layers-
and
this
will
be
a
obviously
a
cluster
operator,
configurable
option
which
layers
you're
interested
in
so,
for
example,
we
were
initially
thinking
about
three
possible
layers.
The
first
one
will
be
identical
to
the
current
one,
which
is
the
HTTP
HTTP
headers
level.
A
So
you
will
basically
get
the
same
information
that
we
currently
provide.
The
next
level
was
request
objects
where
we
would
put
the
entire
object,
as
is
passed,
be
that
part
above
maybe
we
would
probably
and
code
word
above
but
Jason
like
object,
and
that
would
be
unstructured
object.
That
will
be
that
will
be
logged
as
part
as
part
of
the
as
part
of
the
event
and
the
third
level,
which
is
probably
will
be
the
toughest
one
to
achieve,
but
we
would.
A
A
The
rules
remember
how
I
mentioned
that
we
currently
log
everything
that
includes
every
single
user,
every
single
component
and
every
single
request,
so
I've
I've
looked
into
outer
D
in
Linux
and
it
has
similar
notion
of
rules.
The
rules
allow
you
to
filter
the
information
that
you
actually
care
about
in
the
outer
log,
meaning
that
you
are
only
interested
in
certain
calls
for
here.
We
could
limit
the
number
of
the
logging
to
a
particular
user
to
a
particular
resource,
or
maybe
a
particular
namespace,
or
maybe
a
particular
component
that
we're
interested
in.
A
We
would
like
to
have
the
rules
very
composable
and
an
easily
configurable
and
finally,
and
finally,
the
output.
The
current
output
is
log
file
base
right
that
that's
what
I
said.
So
the
best
option
currently
would
be
to
be
able
to
aggregate
all
the
auditing
information
together
into
a
single
place,
from
multiple
API
servers
from
from
cuba,
aggregator
and
from
the
Federated
Federation
server,
all
together
to
a
single
place.
A
For
that
we
we
need
to
be
able
to
obviously
send
that
over
to
a
single
place,
but,
additionally,
we
need
to
be
able
to
correlate
the
those
three
or
multiple
places
together.
Somehow-
and
this
is
where
my
discussion
with
Nikhil
yesterday-
was
about
being
able
to
inject
the
request,
ID
and
and
and
and
pass
it
over
all
the
way
down
through
all
those
layers,
so
that
they
all
can
share
that
information
and
I
will
be
easy
to
to
connect
those
information
afterwards,
so
the
future
is
bright.
A
A
I
want
reach
out
to
me
directly
either
either
personally
I'll
be
here
till
Friday
morning
or
or
any
other
person
that
is
interested
and
I
know
that
Alex
is
is
interested
in
auditing
Stefan
is,
is
going
back
and
back
and
forth
here
somewhere
reach
out
to
us.
We
would
like
to
hear
from
you
if
you're
a
developer-
and
you
have
some
knowledge
around
that,
and
you
can
help
us
with
implementing.
Please
do
so
as
well.
Thank
you
very
much.