►
Description
What are you doing to secure your k8s containers? In this presentation, Conner focuses on the Kubernetes security landscape and presents some best practices aimed at reducing the risk and attack surface of your clusters.
B
A
So
it's
no
surprise
to
anyone
that
I'm
going
to
talk
about
kubernetes
security,
so
first
I'll
just
give
you
a
quick
overview.
We're
gonna
work
on
or
talk
about
today.
Sorry,
it's
just
general
communities,
hygiene
for
security
right,
it's
kind
of
like
a
high-level
and
then
we're
going
to
do
a
deeper
dive
into
a
workload
best
practices
and
I'll
interleave
a
demo
with
some
of
the
best
practices
and
why
they
actually
matter
like
in
a
real,
a
real
in
a
real
world
scenario
of
attacks
and
I
can
take
any
questions
you
guys
might
have
cool.
A
So
what
are
we
all
doing
here
right?
You
know
and
when
I
think
about
kubernetes
right.
Some
of
us
are
pushing
communities
that
are
organizations,
and
some
of
us
are
getting
entities
pushed
on
us
and
our
organizations
right
and
really
what
it
marks.
It's
kind
of
like
a
huge
paradigm
shift
in
infrastructure
and
the
way
that
we
manage
software
and
applications
and
the
way
that
developers
actually
end
up
deploy
their
applications
right.
A
It
makes
me
a
little
scared
right.
You
know,
I
was
I'm
a
developer,
I
still
am
I
worked
in
the
infrastructure
before
I
would
stack,
rocks
and
I.
Don't
think
I
thought
about
security
like
one
time
and
basically
what
I
would
do
is
get
you
know,
develop
code,
make
sure
it
works
on
docker
containers,
push
it
and
hope
for
the
best
and
now
taking
a
security
side
security
view,
it's
probably
not
the
best
way
to
go
right.
So
how
do
we
build
security
into
to
rolling
out
our
software
so,
first
and
foremost
on
communities?
Hygiene?
A
Please
update
right,
make
sure
you
stay
up-to-date.
If
anyone
remembers
communities,
113
deprecates
at
CD,
3.0
right
if
you're
still
on
112
and
a
vulnerability
comes
out,
maintainer
z',
don't
patch.
They
passed
for
the
last
three
release
right
and
the
last
thing
in
the
last
space
you
want
to
be
in
is
having
to
update
your
data
back
plain
while
trying
to
fix
the
mobility
right.
These
things
are
usually
best
thought
through
planned
out.
A
You
know
a
firm
idea
of
a
migration
and
you
don't
want
to
be
stuck
in
this
scenario,
where
you're
trying
to
fix
a
critical
vulnerability,
but
also
rolling
out
tons
of
new
infrastructure
right
I
think
everyone
can
agree.
Infrastructure
upgrades
are
as
scary
as
they
come
and
don't
do
it
quickly
right
so
in.
A
Fall
out
of
the
community
and
see
what's
going
on
right
and
so
I
love
this
Google
group,
they
post
a
ton
and
you'll
get
a
bunch
of
information.
You
know.
Actually
it's
Jack
rocks
as
well.
We
have
like
a
slack
channel
and
then
you
can
post
all
the
news
from
criminales
in
there
right
just
make
sure
you're
up
to
date,
and
you
understand
when
you
see
these
are
coming
out
whether
or
not
they
affect
your
organization
and
what
you
can
do
to
remediate
them.
A
So
more
concretely
like,
let's
make
sure
we
still
use
best
infrastructure
practices
right
like
these,
these
ideas
and
concepts
that
we've
had
for
a
long
time
don't
go
out
the
window.
Let's
make
sure
that
our
network
access
is
far
walled
off
correctly.
Let's
make
sure
our
careers
API
server,
which
is
basically
the
entry
point
until
all
of
your
infrastructure
is
locked
down
and
I
know.
A
A
lot
of
people
will
restrict
it
to
a
VP
C
or
the
other
companies
VPN,
just
to
ensure
that
the
traffic
to
the
API
server
is
protected
and
then,
lastly,
from
like
a
purely
host
perspective,
all
right,
let's
make
sure
that
the
actual
host
itself
is
locked
down.
You
know
Google's
done
a
bunch
of
stuff
with
this
with
costs
right,
they
are
optimized
OS
right
and
in
something.
That's
really
interesting.
A
Clusters
is
to
run
debugging
images
alongside
your
pods
binding
to
the
name
of
the
same
namespaces,
and
you
can
debug
right
there
without
having
all
the
root
privileges
or
having
a
container.
That's
constantly
full
of
tools
available,
right,
I,
think
the
world
where
you
run
one
application
inside
a
container
with
one
process,
is
sort
of
the
dream,
but
that's
been
tough
when
you
need
it
debug
and
actually
like
try
to
test
up
and
see
if
it
still
works.
A
You
know
your
API
server
is
locked
onto
your
VPN,
but
unless
I
make
sure
the
people
who
you
expect
to
use
your
API
server
are
accessing
the
correct
resources.
I
think
I
really
saw
the
way
that
people
have
been
doing.
This
is
through
continuous
deployment
systems
right.
So
you
push
a
lot
of
the
our
back
into
the
continuous
deployment
system,
which
is
usually
hooked
into
some
version
control
right
and
then
you
use
kind
of
your
normal
authentication
through
github.
A
A
A
To
do
that
so
now
you
have
a
much
smaller
number
of
people
you
need
to
manage
in
our
back,
and
you
know:
Kate's
are
back
and
get
a
little
unwieldy.
You
have
to
figure
out
what
all
of
the
verbs
mean,
what
all
the
resources
possibly
exist.
Right
and
all
of
the
subjects
and
Kate's
by
design
has
a
very
open
policy
of
our
Beck.
A
So
that's
kind
of
what
I
have
on
overall
community
security
in
terms
of
like
the
actual
clustering
infrastructure
itself,
and
now
what
I
consider
a
little
bit
more
interesting
is
securing
the
workloads
and
I
actually
think
that
containers
and
current
IDs
provide
kind
of
a
new
way
of
securing
workloads.
Because
now
you
can
look
at
a
single
application
and
scope
down
the
surface
area
to
exactly
what
that
application
uses.
A
A
Namespaces
are
great
for
resource
usage
tracking,
for
seeing
hey,
which
team,
which
tenet
is
using
a
lot
of
resources.
It
allows
our
back
to
be
finely
tuned,
so
you
can
have
a
certain
namespace.
You
can
have
give
someone
just
access
to
those
resources.
It
allows
for
generic
Network
policies
and
segmentation.
I
know
a
pretty
common
use
case
in
the
kubernetes
cluster.
Is
you
have
two
tenants?
You
know
you're
running
a
SAS
model
and
you
don't
want
any
of
the
tenants
to
have
any
connection
to
each
other
right
network
policies.
A
Give
you
a
really
nice
way
to
just
completely
block
between
those
and
then
finally,
like
I,
think
my
favorite
is
make
some
coop
couple
results
a
little
bit
more
sane,
so
you're
not
grepping
and
knocking
through
all
of
the
results
or
trying
to
filter
via
the
right
labels,
and
it
also
makes
it
API
responses
significantly
faster.
So
namespaces
are
great,
so,
in
general,
the
way
I
think
about
risk
in
terms
of
workloads
is
kind
of
in
this.
Like
amazing
hexagon
type
pattern.
A
You
know
it
starts
with
billing
your
image,
depending
looking
at
the
dependencies,
the
packages
analyzing
the
vulnerabilities
right,
because
that's
definitely
a
spot
that
you
need
to
analyze.
So
one
ability
scanning
is
pretty
key
and
looking
at
you
know
the
known
bad,
an
image.
Then
you
look
at
kind
of
how
the
app
is
configured
right
in
terms
of
what
privileges
it
has
on
Linux,
what
privileges
it
has
against
the
API
server
and
form
of
service
accounts
right
and
what
other
sensitive
data
it
may
access
API
keys
for
registries.
A
You
know
sensitive
databases
and
then
actually
on
the
far
right.
The
part
about
labeled
and
annotations
I
think
it's
really
key,
which
is
basically,
how
do
you
loop
back
when
you
look
at
a
service
to
say
who
owns
this
service?
Write
an
annotation
of
owner
or
email
or
team
is
like
super
valuable
and
basically
doing
kind
of
the
debug
cycle
for
operations
and
looking
and
making
sure
that
you
can
route
that
quickly,
as
opposed
to
kind
of
like
the
murder,
mystery
of
trying
to
figure
out
hey.
Who
who
owns
this
application?
A
Who
deployed
this
right
and
you
can
immediately
route
back
to
the
right
people,
so
some
key
configurations
that
I
personally
love
in
containers
are
there
like
read-only
root
file
system,
Linux
capabilities
and
network
policies
and
all
elaborate
and
demo
those,
and
then
there's
like
way
more
than
the
seven
that
I
put
on
this
slide,
but
there's
a
ton
of
configurations
that
can
affect
your
security
posture
so
for
the
read-only
file
system,
it's
pretty
much
exactly
what
it
sounds
like
right.
You
can
make
your
container
filesystem
read-only.
That
means
that
no
one
can
write
to
it.
A
You
can
do
is
create
like
empty
directories
and
volumes,
or
you
can
use
the
volume
command
in
a
docker
file
and
then
that
can
make
that
a
specific
path
readwrite,
where
the
rest
of
the
filesystem
is
read-only
and
the
real
reason
this
matters
is
I,
don't
know
if
you
guys
have
ever
used
Metasploit,
which
is
like
a
penetration
testing
tool.
A
majority
of
Metasploit
like
exploits
end
up,
dropping
payloads
to
slash
temp
right,
and
the
reason
for
that
is
that
traditionally
slash
temp
is
always
writable.
A
A
You
make
a
specific
path,
read,
write
and
the
rest
of
it.
Read-Only
can
easily
like
thwart
an
attacker
who's,
just
running
a
script
against
your
service,
so
the
demo
I
have
and
I'll
just
pray
it
works
and
we'll
see
what
happens
is
I
have
a
vulnerable,
struts
deployment
all
right.
This
is
the
classic
vulnerability
that
equifax
got
hit
with
and
stole
public
it's
sold
well.
A
They
definitely
stole
mine
and
hopefully
not
of
your
social
security
numbers,
but
I'm
just
running
a
strut
server
here
and
then
I'll
run
an
exploit,
and
this
exploit
is
remote
code
execution.
Now
what
we
ended
up
doing
is
on
the
back
side.
It
opens
a
shell.
We
download
a
crypto
miner
right
into
the
filesystem,
make
it
executable
and
execute
it
right,
and
so
now
what
effectively
happens
is
I
have
a
crypto
miner
running
in
my
environment,
which
obviously
sucks
because
they're
stealing
my
resources
right.
A
A
Great
and
I'll
run
the
same
attack
right,
so
in
this
case
the
W
get
call
to
the
minor
D
binary,
failing
because
the
filesystem
is
read-only.
The
next
point
this
kind
of
brings
up
is
like
when
this
is
all
backup
slides.
When
can
I
figure
out,
if
my
app
can
be
read-only
and
what
file
passed
need
to
be
rewrite
right.
It's
not
always
super
easy.
You
know,
for
example,
this
is
engine
X
container.
You
know,
I
didn't
write
engine
X,
I,
don't
know
exactly
what
file
path.
A
A
A
So
I
don't
worry
about
those
too
much
right
and
then,
in
this
case,
what
we
can
see
is
that
var
cache
engine
X
right,
it's
kind
of
like
a
root
filesystem,
and
so
what
I'd
like
to
do
is
I'll,
probably
put
a
empty
directory
at
far
caching
engine
X
right,
because
we
know
we
need
to
write
these
other
five
ten
files
right.
So
in
this
case,
I
can
make
nginx
completely
read-only,
except
for
two
specific
paths.
A
Worked
awesome,
so
the
next
kind
of
area
that
you
can
really
slim
down.
Your
container
is
in
Linux
capabilities,
which
a
lot
of
people
haven't
used,
and
there
they're
definitely
way
more
behind
the
scenes
in
terms
of
containers
and
applications,
because
they
don't
really
have
you.
Don't
really
call
them
directly
right,
like
what
you're
doing
in
your
application
ends
up
calling
Linux
capabilities
and
the
whole
idea
is
basically
to
take
the
route
superpowers.
A
That
route
can
do
everything
and,
let's
split
it
up
into
define
tasks
and,
for
example,
these
here
right
are
kind
of
really
really
common
ones
used.
But
Karina's
gives
you
the
ability
to
drop
capabilities
and
add
capabilities
that
you
need
to
use,
and
so
what
I
did
in
order
to
see
what
Linux
capabilities
the
struts
application
was
using
in
this
case,
this
is,
with
the
minority,
exploit
it
downloads
at
tar.gz
and
then
tarz
it
so
I
used
the
EPB
BPF
probe,
that's
actually
written
by
Brendan
Gregg
from
Netflix
and
what
it
does
is.
A
It
listens
to
a
specific
spot
in
the
kernel
and
looks
for
the
capabilities
that
you're
trying
to
use
right
and
they
can
pull
those
out
for
you.
What
that
can
do,
basically,
is
give
you
like
a
list
of
capabilities
that
you
need
to
use.
You
know
they're,
not
super
easy
to
get
to,
but
they're
just
equally
as
powerful
and
in
stopping
an
attack.
So
in
this
case
this
is
all
you
do.
A
A
Right
so
in
this
case
we
were
able
to
reach
out
to
github
download
the
minor
D,
but
when
they
wanted
to
change
the
ownership
and
then
eventually
exec
create
chmod
dot,
R,
X
or
+
x,
the
binary
they
ended
up
failing
right,
and
so
sometimes
you
can't
always
make
your
file
system
read-only.
But
this
is.
B
A
A
Supposed
to
be
reaching
out
to
the
internet
doesn't
mean
it's.
You
know
it's
only
taking
incoming
requests.
How
do
I
just
block
it
completely,
and
the
answer
is
network
policies.
So
the
way
I
think
about
network
policies
is
that
there
are
l3
l4
firewalls
right
with
some
extra
context
added
for
kubernetes.
So
when
you
look
at
what
this
particular
network
policy
would
do,
the
first
pot
selector
is
selecting
anything
with
the
label
right.
A
So
it's
much
it's
very
generic
and
it
selects
a
web
application
and
then
the
ingress
is
saying
what
can
talk
to
it
right.
So,
in
this
case
you
could
imagine
you
have
Prometheus
running
that
is
scraping
endpoints
in
different
namespaces,
and
what
this
will
do
is
allow
anything
from
the
operations
namespace
of
type
monitoring,
so
that
could
be
a
Prometheus
server
that
we
have
running
and
we'll
be
able
to
scrape
application
web
I'm
just
defining
exactly
that
path
between
that
namespace
and
my
web
application,
and
just
restricting
the
traffic
security
is
hard.
A
So
the
network
policy
that
I'll
apply
to
my
struts
container
is
basically
denying
all
egress
right.
So
my
container
has
the
label
app
and
they
and
the
value
of
API
server
and
the
policy
type
is
egress
now
kind
of
a
pain
with
network
policies.
Is
that
since
I
didn't
supply
any
of
the
rest
of
the
Yambol,
that
means
I
block
everything
which
is
not
always
completely
obvious.
A
Just
taking
longer
than
before,
because
the
network
call
from
W
get
is
hanging
right,
so
we
didn't
never
even
got
to
the
point
we're
trying
to
run
the
minor
D.
We
have
created
a
network
policy
to
block
that
connection
and
wk
is
basically
just
hanging
on
a
connect
to
timeout
and
will
eventually
timeout.
So
this
is
a
way
that
you
can
just
totally
prevent
this
entire
attack.
A
It
has
within
Linux,
and
this
can
give
you
like
a
really
really
good
security
posture
from
a
workload
perspective.
Especially
if
you
can
kind
of
build
us
into
your
developers
workflow,
so
the
last
thing
to
know
always
remember
is
that
security
is
hard
right.
Security
is
like
a
marathon
battle,
all
right,
like
there's,
no
such
thing
as
like
really
perfect
security,
but
it's
always
about
monitoring
iterating
right
and
making
sure
that
the
tools
are
available
for
people
who
are
building
the
code
to
kind
of
integrate
it
into
their
process
and.
B
B
B
A
B
A
On
second
profile
for
a
container
and
you're
wrong,
like
it's
totally
not
gonna
work,
I'm,
almost
a
fan
of
like
pseudo
comp,
where
you
basically
say
like
here's,
the
second
problem
profile,
I
think
it
should
have.
And
then
you
can
validate
events
against
that
and
then
highlight
them
if
they
own
out
of
your
second
profile
to
be
investigated.
A
A
Sure
if
it's
in
1,
1
or
1,
1,
6
I
think
it
might
be
an
alpha.
But
I
was
just
reading
some
stuff
on
it
today
and
I
was
pretty
excited
that
they're
kind
of
moving
that
direction,
because
I
know
that's
been
a
huge
trouble
for
a
lot
of
people
and
you
don't
necessarily
want
to
restart
a
pod
to
add
a
container
that
can
do
debugging
because
then
you've
lost
the
whole
context
of
whatever
you
were
trying
to
debug.
Well,.