►
From YouTube: Webinar: Kubernetes Audit Log - Gold Mine For Security
Description
In the security world, one of the most established methods to identify that a system was compromised, abused or mis-configured is to collect logs of all the activity performed by the system’s users and automated services and to analyze these logs - Kubernetes API Server audit subsystem is exactly that. In an era where operators, controllers. in-cluster components as well as external components are what application made from - having a practice around this security data source becomes a must.
View this webinar to learn about how Kubernetes Audit log can be utilized to enhance the security of Kubernetes based deployments.
A
You
know
so
I.
We
welcome
everybody.
Thank
you
for
joining
for
today's
webinar
for
the
cognitive
computing
foundation.
So
my
my
name
is
alessandro
volta,
I'm
a
principal
engineer
at
Microsoft
and
I'm,
a
CNC
F
ambassador,
so
you're
all
so.
For
today
we're
gonna
have
a
testing
between
are
presented
by
getting
our
CTO
and
co-founder
Alcide,
so
just
a
few
housekeeping
items
before
we
get
started
so,
first
of
all,
unfortunately,
you
cannot
talk
doing
the
webinar,
so
please
just
listen
in
and
there's
a
Q&A
box
at
the
bottom
of
your
screen.
A
So
if
there's
a
there's
any
question
for
a
forgot
e,
please
put
it
there
and
I'm
gonna
read
at
the
end
of
the
webinar,
so
you
have
to
hold
on
until
the
end.
We're
gonna
leave
15
minutes
at
the
end.
To
answer
your
questions
very
important.
This
is
an
official
CN
CF
webinar.
So
we
have
to
adhere
to
the
code
of
conduct
of
the
Carnegie
Foundation.
So
please
refrain
from
using
inappropriate
language.
A
B
Thank
you.
Thank
you,
so
everyone
and
absolutely
thank
you
for
joining
me.
This
is
my
first
webinar
for
the
year,
so
very
excited
to
do
this,
and
in
this
webinar
we
are
going
to
cover
kubernetes
audit
log
and
kind
of
dive
into
why
this
is
a
goldmine
for
security.
So
to
briefly
kind
of
introduce
myself.
My
other
obsession
other
than
kubernetes
and
security
is
cabling.
B
So
just
a
moment
before
I
dive
into
kind
of
kubernetes
audit
log,
and
why
does
it
matter
to
kind
of
look
into
that?
I
want
to
set
the
stage
in
terms
of
like
what
does
it
mean
where
this
piece
of
the
puzzle
falls
in
terms
of
the
entire
security
of
our
kubernetes
clusters,
so
in
a
normal
kind
of
constellation,
we
will
see
our
pipelines
CI
and
CD,
and
these
are
kind
of
two
completely
beasts.
Normally
with
CI
we
would
do
the
source
scanning
and
then
we
will
build
them
into
containers.
B
Normally,
we
will
do
that
using
subsystems
like
hell
or
terraform
or
combination
of
scripting
and
templating,
but
at
the
end
of
the
day,
we
pretty
much
deploy
those
assets
into
our
get
cluster
and,
at
this
point
of
time
and
I
will
touch
that
briefly
and
later
the
resources
go
through.
What
kuben
in
kubernetes
language
is
the
admission
phase.
It's
pretty
much
allows
to
kind
of
inspect
the
resources
before
they
are
introduced
in
the
cluster.
B
B
And
this
would
this
is
going
to
be
the
focus
of
this
session.
So
if
we
kind
of
take
a
deeper
step
into
what
this
kubernetes
api
server
means,
so
essentially
the
API
server
is
kind
of
the
brain,
or
at
least
a
very
important
piece
of
the
brain
of
the
cluster.
So
if
you
think
about
the
kubernetes
control,
plane
Trane's
the
cluster,
every
request
that
is
being
submitted
to
the
API
server,
it
basically
goes
through
kind
of
four
stages.
B
The
first
one
is
pretty
much
the
authentication
we
want
to
establish
the
identity
of
whoever,
in
that
perspective,
the
principle
that
actually
would
like
to
perform
an
action
or
request
against
the
cluster.
At
this
point,
the
request
processing
will
switch
over
to
the
authorization
stage.
So
essentially,
at
this
stage
we
determine
whether
the
established
identity
or
whoever
is
trying
to
perform
an
action
against
the
cluster
is
allowed
to
perform
this
action.
B
This
is
specifically
something
that
we
configure
using
all
those
auerbach
configuration
as
part
of
our
provisioning
of
the
cluster
resources
themselves,
and
if
we
pass
this
gate,
the
resource
is
basically
submitted
to
the
admission
controlling
phase
where
we
basically
perform
two
main
actions
within
this
stage.
First,
one
is
kind
of
mutating,
so
there
are
two
types
of
admission
controllers.
We
are
mutating
admission
controllers
that
can
essentially
change
the
resource
that
we
pretty
much
admit
into
the
cluster.
One
good
example
here
would
be,
for
example,
in
Easter.
B
The
injection
of
the
sidecar
is
something
that
is
being
done
using
a
mutating
admission
controller,
if
you
think
about,
for
example,
tooling,
like
like
secret
injection
and
in
such
subsystems,
you
can
implement
those
using
mutating
admission
controllers.
Once
we
pass
this
gate,
we
go
through
the
validating
admission
controllers,
which
pretty
much
can
say,
yay
or
nay,
based
on
compliance
checks,
hygiene
checks
or
any
policy
that
we
would
like
to
kind
of
apply
to
the
final
version
of
the
resource,
and
once
we've
done
that
the
resource
will
go
through
kind
of
a
kubernetes
validation
stage.
B
Where
would
pretty
much
validate
that
the
fields
and
everything
kind
of
conformed
to
the
expected
layout
of
the
resource?
And
from
this
point
onward?
This
is
something
that
is
being
admitted
into
the
cluster.
Now
the
API
calls
it
basically
performs
the
entire
kind
of
machinery
that
I
described
here
basically
is
basically
captured
by
the
API
server.
B
So
if
we
try
to
kind
of
ask
ourselves
like
who
needs
access
to
who
actually
access
the
API
server,
we
can
see
that,
because
kubernetes
inherently
or
pretty
much
natively
encompass
the
notion
of
of
operators
and
controllers
and
control
loops,
then
we
can
pretty
much
break
it
down
to
kind
of
three
main
pieces
or
groups.
The
first
one
is
kind
of
human
operators.
These
are
users
access
the
cluster.
B
Normally
this
would
be
either
using
specially
written
kubernetes
clients
using
the
client
libraries
that
the
kubernetes
SDK
offers,
or
it
can
be
using
the
queue
control
command
line
or
whether
this
is
helm
that
is
deploying
into
the
cluster
itself.
The
second
group
is
basically
captured
by
the
system
component,
so,
for
example,
the
crew
blades
that
are
running
on
the
nodes
and
represent
kind
of
the
kubernetes
agents,
repeatedly
or
kind
of
continuously
will
probe
the
API
server.
B
For
example,
once
they
get
in
a
request
to
schedule
a
pod
on
the
node,
the
couplet
will
fetch
from
the
API
server
the
config
maps
and
the
secrets
that
are
relevant
to
this
port.
So
there
is
an
ongoing
access
being
performed
by
the
system,
components
to
pretty
much
maintain
kind
of
the
desired
state
or
the
expected
state
of
the
resources.
Other
good
example
would
be
every
deployment
that
we
push
into
the
cluster.
B
That
kind
of
in
in
a
cascading
event,
will
fire
up
a
replica
set
and
the
pods
and
the
underlying
pods
are
also
accessing
the
cluster
using
special
kind
of
designated
group
when
we
are
looking
at
this
kind
of
from
the
API
server
perspective
and
the
last
kind
of
component
or
group
is
basically
service
accounts.
These
are
service
account
that
we
provision
as
part
of
the
deployment
of
resources
where
we
grant
certain
workloads
certain
permissions
to
do
against
or
to
perform
against
the
API
server.
B
One
good
example
would
be
for
the
operators,
so,
if
you
think
about
Prometheus,
for
example,
this
is
normally
kind
of
a
privilege
component
when
I'm
saying
privilege
it's
kind
of
with
respect
to
which
api's
it
allows
to
kind
of
access.
The
API
server-
and
these
are.
This-
is
a
type
of
other
resources
that
require
access
from
the
API
server.
B
We
have
kind
of
a
very
large
stream
of
data
that
is
pretty
much
policy
driven,
so
you
can
tweak
which
kind
of
data
flows
in
those
event
or
how
verbose
those
logs
can
be,
and
we
pretty
much
from
a
security
standpoint
would
like
to
surface
signals
that
are
meaningful
and
actionable
and
we
can
use
in
our
day-to-day
kind
of
security
operations
of
the
cluster.
So
one
of
the
main
challenges
is:
how
do
we
crunch
this
raw
data
into
meaningful
insights
inside
the
cluster?
B
So,
for
example,
and
we'll
kind
of
dive
into
that
deeper
later
on
is?
Is
someone
exploiting
a
vulnerability
in
my
80
server?
It
can
be
a
known
API.
It
can
be
unknown
vulnerability
in
the
API
server
because
we
didn't
upgrade
the
server
in
kind
of
in
time,
but
it
can
be
unknown
vulnerability
that
pretty
much
leaves
signals
or
traces
inside
the
audit
love
itself.
It
can
be
scenarios
where
stolen
credentials
from
users
or
stolen
tokens
from
service
accounts
are
being
exploited
or
used
or
abused
inside
the
cluster.
B
We
want
to
take
those
we
have
instances
of
kind
of
compliance
based
checks,
I
mean
if
someone
I'm
kind
of
exact
into
a
pod.
This
is
something
that,
for
example,
an
auditor
or
compliance
from
a
compliance
standpoint
would
like
to
keep
track
of,
and
another
example
would
be
if
a
component,
basically
at
the
Miss
configuration
in
the
arbok.
This
is
something
that
leaves
traces
inside
the
audit
log
itself.
B
So
when
we
try
to
think
about
how
do
we
get
kind
of
the
audit
stream?
This
is
something
that
kind
of
is
relatively
a
moving
target
when
it
comes
to
the
different
versions
of
kubernetes
that
you
are
using.
So
the
kubernetes
native
kind
of
approach,
which
is
still
an
alpha
level,
api
t2,
it
was
introduced
in
in
version.
One
point:
thirteen
is
the
ability
to
stream
the
audit
log
to
audit
sinks
that
reside
inside
the
cluster.
B
It's
a
new
resources
that
basically
enables
to
register
an
audit
log
target
and
the
kubernetes
api
server
will
kind
of
stream
the
audit
log
to
this
API
web
to
this
audit
web
hook.
So
essentially,
it
enables
us
to
kind
of
plug
in
an
inspection
point
of
the
audit
log
that
resides
as
part
of
the
cluster.
B
So
you
can
think
about
it
as
something
that
you
want
to
build
to
employ
security
analysis
or
or
Mis
configuration
analysis
as
part
of
your
cluster,
the
infrastructure
and
if
we
take
kind
of
few
more
examples
from
from
some
of
the
cloud
providers,
so
in
in
gke,
for
example,
by
default,
there
is
an
audit
policy
in
place
that
basically
stream
the
audit
log
to
stag
driver
which
is
kind
of
the
built-in
I'm
logging
log
shipping
service.
In
aks.
B
You
can
leverage
the
event
hub
to
basically
extract
the
kubernetes
audit
log
for
external
processing
and
in
AWS
eks.
We
pretty
much
leverage
or
enable
kind
of
EK
has
to
ship
those
audit
logs
to
cloud
watch
and
then
by
streaming
them
to
Kinesis.
We
can
pretty
much
fetch
those
logs
in
order
to
analyze
them.
B
So
what
those
audit
logs
actually
look
like
in
practice,
so
this
is
kind
of
a
trimmed-down
version.
I
didn't
put
all
the
fields
that
basically
are
part
of
the
audit
log,
but,
as
I
mentioned
earlier,
kubernetes
has
an
audit
policy
that
basically
controls
the
verbosity.
What
exactly
exists
in
each
and
every
one
of
the
logs,
so
you
can
basically
control,
for
example,
whether
the
resource
itself
will
be
placed
in
line
as
part
of
the
log
on
it.
B
The
audit
log
itself,
so
one
one
bad
example-
would
be
to
configure
the
audit
policy
to
place
the
objects,
the
resource
centers
themselves
of
type
secret.
It's
pretty
much
would
be
revealing
your
kubernetes
secret
resources
inside
the
audit
log
and
if
you
ship
it
to
kind
of
external
systems,
you
pretty
much
created
kind
of
situation
where
you
leak
secrets
in
a
similar
fashion,
where
why
you
don't
want
to
leak
secrets
in
kind
of
the
console
logs
of
your
container
or
or
pod.
B
So
essentially,
when
we
try
to
think
about
what
exactly
the
components
that
we
have
in
the
other
long.
So,
first
of
all,
we
have
the
resource
type
that
a
principal
or
our
service
account
is
trying
to
access,
and
then
we
have
assuming.
This
is
a
namespaced
resource.
We
have
the
name
gist
of
the
resource
and
then
we
also
get
to
see
which
verb
HTTP
verb
in
for
that
matter.
B
It's
pretty
much
represents
whether
we
are
reading
whether
we
are
listing
whether
we
are
deleting
or
updating
the
resource,
and
then
on
top
of
that
we
have
a
few
kind
of
pieces
of
information
that
are
extremely
valuable.
The
first
one
would
be
kind
of
the
user,
the
established
user
identity
as
part
of
the
authentication
process.
B
So
essentially
we
can
see
here
the
user
name
and
the
group
that
the
user
is
kind
of
part
of
and
when
we
try
to
think
about
it
from
establishing
profiles
and
and
understanding
kind
of
the
nature
of
API
operations
being
performed
against
the
cluster.
These
are
very
important
features
to
track
when
we
try
to
understand
and
of
who
is
doing
what
and
figuring
out
this
kind
of
entire
picture.
We
have
a
couple
of
more
items
or
or
fields
in
the
audit
log.
B
The
user
agent
basically
represents
the
client,
something
that
identifies
the
client
version
that
connects
to
the
kubernetes
api
server.
So,
for
example,
if
I'm
you
know,
if
I'm
accessing
my
kubernetes
cluster
from
let's
say
the
Google
dashboard,
then
essentially
we
will
see
kind
of
the
user
agent
being
embedded
by
the
Google
kind
of
API
client
as
Google
container
engine.
On
top
of
that,
we
will
see
network
information
or
network
location
from
which
the
API
invocation
was
made.
Some
of
the
fields
that
we
see
here
can
be
spoofed
by
an
attacker.
B
Whether
this
is
something
that
I
was
denied
by
the
API
server
and
in
a
lot
in
the
rest
of
the
field.
You
can
pretty
much
see
like
error,
codes
and
and
kind
of
more
specific
data
that
we
can
derive
some
some
insights.
Just
to
give
you
a
sense
like
so
something
that
was
in
the
recent
cube
con.
They
do
get
a
really
nice
talk
on
kubernetes
audit
log
and
how
they
basically
monitored
the
audit
log
internally
and
on
a
2500
cluster.
B
The
amount
of
audit
logs
per
per
minute
was
was
1,000
1,000
events
per
second,
so
1,000
events
per
second.
So,
if
you
think
about
it,
this
is
relatively
a
very
large
amount
of
data
that
needs
to
be
processed
and
digested
and
because
there
are
very
important
signals
that
actually
reside
in
the
data.
It
relatively
requires
some
some
heavy
lifting,
or
at
least
some
analysis
to
basically
surface
those
insights
from
those
logs.
So
let's
take
a
few
use
cases
as
an
example.
B
Here,
from
a
troubleshooting
perspective,
we
can
pretty
much
detect
system
failures
based
on
error
code,
for
one,
for
example,
from
nodes.
So
specifically,
one
example
would
be
if
cubelet
on
one
of
the
nodes
cannot
connect
to
the
API
server
and
we
start
to
see
kind
of
a
drift
in
the
number
of
error
codes
that
we
see
from
resources
that
are
part
of
the
system
nodes.
B
Then
this
is
something
that
we
can
pretty
much
fire
to
the
sick
to
the
SRE
team
to
take
down
the
nodes
or
drain
them
and
make
sure
that
they
can't
connect
to
the
API
server.
Another
thing
that
I'll
s
kind
of
would
recommend
from
a
practical
standpoint,
would
be
to
check
the
responsiveness
of
the
API
server
by
measuring
what
is
the
in
fly
time
or
the
mean
time
to
response
time
of
the
API
server.
Calls
that
you
that
we
have.
B
Let's
take
another
example
from
more
of
a
security
kind
of
standpoint.
We
think
about
our
production
clusters
is
kind
of
the
closed
environment,
well,
gated,
environment.
So
one
of
the
kind
of
interceptions
or
compliance
kind
of
elements
that
we
can
plug
into
the
audit
log
in
our
systems
is
the
ability
to
trigger
alerts
when
sensitive,
pods
or
in
general.
Even
all
the
pods
in
certain
environments
are
being
accessed
some
some
use
useful
kind
of
information
around.
B
B
Those
events
in
the
right
time
and
kind
of
put
them
aside
for
an
audit
or
to
kind
of
inspect
another
example
would
be
if
you
want
to
have
some
internal
security
guard
rails,
built
into
our
environments,
where
production
clusters
are
not
being
either
exact
or
proxied
into
a
pods
or
even
dumping
logs
that
are
that
may
contain
sensitive
data.
Then
this
is
something
again
that
we
can
pretty
much
leverage
the
audit
log
to
help
us
it,
and
it's
very
useful
to
kind
of
use
that,
because
it
covers
a
lot
of
grounds.
B
From
that
perspective,
another
kind
of
deeper
example
would
be
to
kind
of
detect
miss
configuration
in
our
environment,
so
any
kind
any
kind
of
unauthorized
access
to
the
cluster
that
basically,
is
represented
by
a
permission
denied,
for
example,
return
code
from
the
API
server
is
something
that
we
can
track
and
pretty
much
based
on
that.
We
can
understand
whether
we
have
kind
of
errors
in
our
environment
or
miss
configuration
or,
for
example.
B
This
is
something
that
represents
kind
of
malicious
activities
against
our
API
server,
so
they
are
always
kind
of
in
with
that
respect,
and
normally
it
can
be
associated
with
miss
configuration.
But
on
the
other
hand,
this
can
be
also
someone
kind
of
a
threat
actor
that
may
be
inside
our
cluster
or
breached.
One
of
our
DevOps
is
trying
to
do
operations
that
are
not
pretty
much
part
of
the
permissions
that
the
breached
component
is
allowed
to.
B
This
is
an
unusual
kind
of
way
to
work
with
your
cluster
I
mean
the
idea
here
is
that
we
can
detect
those
events
by
analyzing
the
audit
log,
looking
at
the
source
IP
addresses,
for
example,
that
exist
in
the
audit
log
and
by
tracking
those.
We
can
understand
whether
this
is
something
that
kind
of
falls
into
this
category.
B
So
we've
talked
a
lot
about
what
we
should
do
with
our
audit
log,
but
some
of
the
elements
that
audit
log
doesn't
give
us
so
the
first.
The
first
thing
is:
if
we
want
to
check
the
hygiene
of
the
resources,
for
example,
you
want
to
make
sure
that
we
don't
have
towards
that
are
running
as
privileged
components,
or
we
want
to
make
sure
that
in
our
config
maps
we
don't
see
any
API
access,
keys
or
tokens
or
password
or
anything
of
that
sort.
B
So
the,
in
my
opinion,
the
wrong
kind
of
way
to
approach
that
is
to
configure
your
API
server
to
basically
dump
the
resources
themselves
into
the
log,
the
audit
log
and
running
those
inspections
through
the
audit
log.
There
are
much
simpler
mechanisms
to
perform
that
admission
controllers
would
be
one
way
of
doing
that,
or
even
just
calling
the
API
server
reading
the
resource
or
dumping
the
entire
resources
and
running
the
scan
against
them.
B
The
main
reason
for
that,
when
you
have
kind
of
a
growing
number
of
nodes
in
your
cluster
or
a
growing
number
of
controllers
in
your
clusters
and
then
shipping
all
the
resources
themselves,
as
all
of
those
kind
of
automated
components,
read
and
write
stuff
to
the
API
server,
you
pretty
much
move
a
lot
of
data
from
kubernetes
server
to
it.
Kind
of
targets-
another
example
which
I
kind
of
highlighted
earlier
performance
monitoring.
B
We
want
to
use
kind
of
the
metrics
that
are
available
and
exported
by
the
API
server
itself
through
the
metrics
endpoints
and
leverage
Prometheus,
an
alert
manager
or
any
monitoring
solution
that
you
have
there
to
basically
understand.
If
there
are
performance
related
issues
in
in
our
environment,
you
can
use
audit
log
to
do
kind
of
the
second-tier
troubleshooting
of
issues,
just
like
we
kind
of
went
over
with
system
nodes
being
unable
to
connect
to
the
API
server
itself.
B
So,
for
example,
if
there
is
a
pod
restart,
it's
not
tracked
by
the
API
server
logs
or
the
audit
log
for
that
matter,
it
actually
covered
by
the
event
subsystem
that
basically
captures
lifecycle
events,
and
this
is
not
something
that
is
covered
by
the
API
server
audit
log.
So,
let's
try
to
take
a
look,
I'm
gonna,
be
brave
here
and
try
to
see
if
we
can
connect
to
a
real
system
that
kind
of
monitors,
one
of
one
of
publicly
facing
clusters,
and
naturally
it
starts
with
a
snap.
B
So
I
hope
the
demo
gods
will
be
with
me
all
right.
So
what
I
would
like
to
show?
You
is
an
example
of
real
kind
of
life
scenario
where
this
is
an
example
here,
where
the
timeline
that
we
see
here
pretty
much
cap
choose
who
access
the
cluster
at
what
point
of
time
and
which
principle
basically
access
the
environment.
B
This
is
kind
of
a
very
easy
and
go-to
kind
of
dimension
to
look
at
the
audit
log
from
which
countries
are
kubernetes,
cluster
is
being
accessed
or
at
least
attempts
to
access
the
cluster.
So
if
you
have
any
kubernetes
api
server
that
is
publicly
or
facing
kind
of
the
internet,
so
if
you're
pretty
much
using
the
default
on
a
KSDK
or
eks,
that's
by
default
kind
of
the
the
reality.
So
you
should
know
that
your
API
servers
are
being
probed
constantly
by
internet
scan,
or
some
of
them
are
actually
tailor
made
for
kubernetes.
B
Now
the
nice
thing
about
this
I
can
basically
pivot
or
pretty
much
focus
the
system
on
the
individual
kind
of
access
attempts
being
made
from
France,
and
you
can
see
that
once
I
filtered
out
the
entire
audit
log
into
kind
of
the
dimension
of
an
IP
coming
from
France,
we
can
see
that
that
very
same
IP
was
accessing
the
API
server.
Specifically,
this
was
last
night
and
the
API
server
basically
responded
with
permission.
B
Denied
I
can
also
try
to
see
and
pretty
much
figure
out
that
the
origin
of
this
access
attempt
was
trying
to
access
the
API
endpoint
to
read
pods,
and
we
can
see
that
the
action
itself
is
basically
captured
by
the
verb
that
we
pretty
much
saw
in
the
audit.
Look
so
crunching
the
data
or
the
audit
log
stream
and
breaking
it
down
to
different
dimensions
and
enriching
them.
B
Pretty
much
enables
us
to
see
or
pretty
much
rule
out
and
understand
if
there
is
something
wrong
going
on
in
our
cluster
or
if
everything
is
kind
of
in
line
with
certain
assertions
or
expectations
about
our
environment.
Just
to
give
you
kind
of
another
example,
we
can,
for
example,
see
based
on
the
audit
policy
who,
for
example,
perform
exit
operations
into
the
cluster
itself.
B
We
can
track
it
down,
so
you
can
see
that
some
of
our
automation
system
is
basically
accessing
the
system
for
regular
kind
of
operations,
but
because
we
pretty
much
captured
those
kind
of
rules
in
our
policy,
we
can
pretty
much
fire
alerts
or
events
to
a
slack
channel
or
whatever
security
tooling.
We
have
there
to
basically
capture
those
events,
because
this
is
something
that,
as
an
internal
practice,
we
make
sure
to
kind
of
track
and
respond
to.
B
So.
With
that
in
mind,
just
to
kind
of
conclude
what
we
covered
here,
cumin
are
kubernetes.
Audit
logs
is
pretty
much
that
the
logs
are
incredibly
valuable
for
both
apps
and
security,
and
when
we
try
to
think
about
what
we
can
do
with
them,
we
can
take
a
lot
of
advantage
of
those
logs,
but
it
does
require
some
effort.
We
saw
that
we
can
pretty
much
stream.
B
The
audit
log
into
kind
of
an
analysis
pipeline
so
streaming
them
or
or
acquiring
the
data
source
is
the
effort
around
that
at
this
point
in
time,
is,
is
not
straightforward,
with
all
the
managed
kubernetes
services
that
are
out
there.
But
on
the
other
hand,
this
is
something
that,
over
time,
once
the
api's
are
maturing
and
will
get
kind
of
2v1
that
this
should
be
relatively
easier
to
kind
of
handle.
B
But
I
will
say
that
crunching,
the
data
and
performing
the
analysis
will
require
using
some
tailor-made
tooling.
It
understands
both
kind
of
static
exceptions
or
rule-based
exceptions
to
whatever
is
flowing
in
the
cluster,
but
detecting
anomalies
and
understanding
kind
of
deviations
from
from
base
profiles,
and
all
that
is
something
that
requires
some
some
AI
to
build
and
bake
into
the
analysis
stream,
and
the
other
kind
of
element
is
that
we
want
to
make
sure
that
our
audit
policy
balance
the
verbosity
of
whatever
is
happening
inside
the
cluster.
B
The
audit
log
is
an
extremely
robust
kind
of
channel
or
data
source,
and
we
want
to
make
sure
that
we
kind
of
tune
the
policy
to
make
sure
that
we
cover
all
of
them.
The
the
use
cases
the
the
audit
log
can
cover
from
both
ops
and
security
perspective
are
extremely
wide.
They
can
address
both
kind
of
regulated
environments
like
like
PCI
or
HIPAA,
or
similar
kind
of
external
regulations,
but
just
as
well
create
some
internal
policies
and
follow
them
and
implement
them.
B
Using
the
audit
log,
you
can
kind
of
detect,
known
and
unknown,
exploits
in
the
API
server
through
monitoring
and
applying
kind
of
higher
level
analysis
of
the
logs
themselves.
So
this
is
kind
of
a
very
valuable
source
of
information
for
detecting
kind
of
such
breach
attempts
or
even
successful
breach
attempts
and
all
in
all,
we
highly
recommend
kind
of
employing
this
data
source
is
part
of
your
security
tool
stack
to
cover
this
so
I
think
it
pretty
much
wraps
up
what
I
wanted
to
kind
of
share
on
the
kubernetes
audit
flog.
B
You
can
register
for
early
access
to
the
audit
log
I'll
sit
k,
audit,
which
basically
will
give
you
some
of
the
dashboards
that
we
saw
earlier,
or
you
can
try
our
cloud
service
to
do
kind
of
some
of
the
other
community
security
challenges
that
are
out
there
and
I
think
that
the
debt.
But
at
this
point
we
can
switch
over
to
some
questions
from
the
audience.
A
Thank
you.
Thank
you,
yeah.
Thank
you
for
the
webinar
was
a
illuminating.
I
also
have
a
few
new
things
about
all
the
logs
I
never
stopped
learning
what
there
there
are.
A
couple
of
questions
that
we
threw
in
chat,
one
is
from
Maury
CEO
is
asking.
If
this
panel
is
native
from
kubernetes,
who
did
the
dashboard,
you
showed
an
ad
from
kubernetes
all
you
have
to
install
something
know.
B
So
the
dashboard
that
I
saw
is
actually
kind
of
an
early
version
of
CK
audit.
So
this
is,
if
you
want
to
get
that,
then
you
should
hit
the
link
the
first
link
and
get
that
there
is
I,
think
that
there
are
some
open
source
tools
out
there,
even
a
CN
CF
on
Falco
that
basically
will
able
to
plug
kind
of
the
audit
stream
and
do
some
of
the
basic
rules
that
are
out
there,
but
the
more
advanced
of
the
AI
stuff.
A
Okay,
thank
you.
A
bunch
of
questions
are
popping
up
now.
So,
let's
start
with
a
breeze.
So
if
a
breeze
is
asking,
if
say
a
Michael
of
java
microservices
with
a
reverse
shell,
vulnerability
was
deployed
on
node
I
will
destroy
the
tech
when
an
intruder
gets
unauthorized
access
with
the
node.
So
so
you
get
the
question.
So
let's
say
that
you
are
born
in
voluntary
or
deploying
unsecure
application.
A
B
B
So
if
the
attacker
will
try
to
kind
of
do
the
lateral
movement
inside
the
cluster
to
where
my
kind
of
assets,
my
data
assets
are
through
attempts
to
call
the
API
server
so,
okay,
all
that
will
basically
detect
that
because
this
would
be
kind
of
captured
using
the
AI
and
the
anomaly
detection.
However,
this
is
why
I
kind
of
in
my
first
it
was
the
first
slide
in
this
webinar
I
mentioned
that
the
runtime
defenses
require
different
type
of
security,
tooling
or
a
security
tool
to
cover.
B
So
the
network
security
is
not
covered
by
the
audit
log.
So
essentially,
if
you
want
to
cover
kind
of-
or
at
least
if
you
want
to
track
down
or
hand
down
an
attacker,
a
sophisticated
attacker
that
will
kind
of
use
multiple
channels
to
do
the
lateral
movement
and
exfiltrate
data,
you
will
need
to
have
more
than
one
tool
in
your
cluster.
A
B
I
would
guess
that
that
when-
and
this
is
something
we
change-
the
cluster
I
mean,
the
cost
is
pretty
much
a
living
creature.
I
mean
if
you
deploy
a
CRD,
for
example,
and
it
creates
a
chain
of
events,
then
the
best
kind
of
the
best
way
to
kind
of
approach
that
is
is
for
example,
if
you
want
to
monitor
or
detect
disruptions
that
are
associated
with
miss
configuration,
you
want
to
track
down
the
permission
denied
kind
of
the
4-3
error
code
from
the
API
server.
B
So,
for
example,
if
your
steady
state
represents
kind
of
0
for
3
in
your
cluster,
you
did
kind
of
a
new
deployment
and
now
before
3
spiked,
then
this
is
something
that
we
would
kind
of
flag
you
to
see
what
happened
in
the
last
deployment
and
whether
this
is
a
configuration
error.
You
can
go
to
the
audit
log
itself
and
look
specific
audit
entries
that
has
the
403
error
code
and
then
look
at
their
actual
resources
that
fire
them.
B
So
that
would
be
kind
of
one
motion
of
leveraging
the
the
audit
log
for
analysis,
but
it's
not
part
of
so
the
mechanics
or
the
machinery
around
that
is
I
would
say
it's
it's
pretty
much.
You
need
to
tailor
use
cases
to
kind
of.
If
you
want
to
do
this
tactical,
if
you
employ
employee
kind
of
AI
into
that
a
good
day,
I
will
detect
those
anomalies
with
relatively
low
low
kind
of
false
positive
rate.
A
B
A
B
So
we
have
some
security
metrics
that
are
exposed
from
ke
audits
it
just
you
need
to
kind
of
enable
them
by
default.
We
don't
enable
the
security
matrix
pretty
much,
because
this
closing
kind
of
security,
I'd
say
it's
pretty
much
disclosed-
is
kind
of
aspect
of
the
security
posture
of
your
environment,
so
we've
kind
of
sensitive
or
paranoid
about
that.
A
B
We
don't
actually
persist
the
entire
audit
log.
Luckily,
for
us
we
just
store
a
digested
version
of
the
audit
logs,
so
the
entire
row,
or
it
alone,
is
actually
stored,
you're
normally
that
would
be
or
cloud
by
the
so
stackdriver
cloud
watch
to
s3
or
aks
to
an
azure
blob,
whichever
kind
of
mechanism
you
do,
however,
crunching
the
data
is
something
that
we
kind
of
persist
on
the
using
actually
stateful
set
to
persist
it
on
the
node
itself
and
if
you
use
kind
of
a
network
attached
storage.
So
it's.
B
Great
question
so
one
of
the
main
kind
of
differences
that
that
chaotic
basically
brings
into
the
table
it.
It
was
built
like
tailor-made
analysis
of
the
kubernetes
audit
log.
It
pretty
much
captures
or
understands
some
basic
or
core
concepts
of
kubernetes
itself
like,
for
example,
what
are
nodes
in
the
cluster
or
what
are
service
accounts
or
what
a
user's
or
how
the
api's
are
being
structured.
B
What
is
resource
and
the
way
that
the
analysis
is
being
done
is
that
we
pretty
much
build
profiles
for
each
and
every
user
in
the
cluster
in
each
and
every
resource
in
the
entire
cluster
as
a
whole,
and
when
you
look
at
kind
of
generic
tools
like
like
elastic
or
Splunk,
they
would
do
kind
of
general
kind
of
machine
learning
analysis
that
normally
would
yield
more
false
positives.
If
at
all
versus
you
know
something
that
is
more
tailor-made
for
for
kubernetes.
A
B
B
A
A
B
B
B
A
B
See
any
reason
why
not
to
I
mean
we
are
pretty
much
kind
of
think
of
us
as
kind
of
security
infrastructure
app
inside
your
cluster
and
if
you
can
migrate
any
other
application
I
mean
Valera
will
basically
take
care
of
exporting
your
volleyed
data
volumes
and
you're
pretty
much
resources.
So
that
should
be
straightforward.
A
Yeah,
that's
what
I
suppose
as
well
so
well,
this
kind
of
egg
exhausts
all
the
question.
If
there's
any
really
last
minute,
question,
please
stop
it
now.
Otherwise,
I
will
I
will
thank
our
speakers
for
today
and
it's
very
good
job
and
it
was
great
to
learn
a
few
more
things
about
how
did
logs
and
I'm
looking
for
I
sign
up
for
the
for
the
for
the
preview,
so
I
hope
you.