►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
This Year, It’s About Security - Maya Kaczorowski & Brandon Baker, Google
The message was resoundingly clear at KubeCon EU, “this year, it’s about security”. Kubernetes has made giant strides in 2018 to improve security for end users.We’ll start with an overview of what’s happened in 2018, including the first container security attacks. Then, we’ll cover focus on three hot topic areas to dive deeper and demo: (1) Isolation, using projects like gVisor, Kata containers, and Nabla; (2) Software supply chain security, and (3) Security by default and hardening. You’ll leave with an understanding of new security features in Kubernetes, and how you can contribute to making Kubernetes secure.
To Learn More: https://sched.co/GrZo
A
So
the
purpose
of
this
talk
is
to
talk
about
security,
and
earlier
this
year,
at
coop
Con
in
Copenhagen,
one
of
the
messages
in
the
keynotes
was
resoundingly
clear
this
year.
It's
about
security.
Now
that
we're
here
in
Seattle-
and
it's
been,
you
know
it's
becoming
a
reality,
we're
seeing
in
the
last
couple
of
months,
we've
seen
huge
strides
to
improve
security
and
kubernetes,
and
so,
let's
look
back
at
what's
actually
changed
in
this
talk.
A
We'll
cover,
what's
happened
in
2018
in
Korea
security,
cool,
new
features
that
are
worth
a
second
look,
including
isolation
and
software
supply
chain
and
hardening,
and
then
what
we
hope
to
see
in
2019.
So,
let's
jump
in.
Let's
start
with
the
worst
of
what
happened
in
2018.
What's
actually
happened,
that's
affected
a
security
in
this
space.
A
Looking
back
at
the
year
in
threats,
we
haven't
actually
seen
that
many
active
attacks
in
the
wild-
it
seems
the
industry
is
still
just
picking
up
in
this
regard,
as
attackers
learn
about
containers
and
cribben
any's
and
how
they
can
best
exploit
these
in
February
Tesla
made
the
news
for
attackers
finding
and
accessing
for
Communities
dashboard
and
using
the
cloud
credentials
that
they
gained
to
mine
cryptocurrency.
This
was
similar
to
something
that
happened
in
December
of
2017,
with
Aviva
in
May
Shopify
had
a
researcher
report
of
owner
ability
in
their
cluster
configuration.
A
This
was
not
exploited
in
June,
Weight
Watchers
had
the
exact
same
problem
that
Tesla
had
it
was
found
by
a
researcher
once
again
and
not
exploited
and
the
same
month,
several
container
images
were
found
on
docker
hub
that
had
embedded
malware
for
mining
cryptocurrency.
Anything
that's
happened
more
recently.
Unfortunately,
we
just
don't
know
about
yet
the
NSA
does
what's
really
interesting
about
all
these
attacks
you
might
notice
is
that
none
of
them
explicitly
attack,
containers
or
container
eyes
workloads.
Some
do
attack
kubernetes
and
crannies
configurations,
but
the
end
goal
isn't
even
kubernetes.
A
It's
something
more
interesting
because
the
ads
attack,
that's
not
how
an
attacker
can
most
easily
make
money
by
mining
cryptocurrency.
Let's
look
at
each
of
these
a
little
bit
more
so
starting
with
Tesla
and
WeightWatchers,
because
what
happened
here
is
essentially
the
same
thing:
if
you're
not
familiar
with
it,
the
criminals
dashboard
is
a
web-based
interface
to
interact
with
your
communities
deployment
it
lets.
You
see
what
your,
what
your
deployment
is
doing.
Look
at
the
state
of
resources
in
your
cluster,
any
errors
that
might
have
occurred,
etc.
A
Now,
if
you're
not
familiar
with
it,
it's
probably
because
you're
using
a
cloud
hosted
kubernetes
solution
from
a
different
from
a
provider.
They
probably
have
their
own
dashboard,
their
own
console
that
you're
using
that's
why
you
haven't
seen
this
before.
So
what
are
hackers
doing
with
this
dashboard?
There
are
two
main
faux
pas
to
avoid
here.
The
first
is
having
the
dashboard
be
publicly
accessible,
so
that
it's
not
really
bad,
that
it's
a
public
IP
address
as
long
as
you
actually
protect
access
to
that.
A
This
prevents
anyone
from
just
going
in
and
looking
at
your
cluster
meddling
with
your
configurations,
etc.
But
actually,
when
attackers
are
exploiting
a
dashboard,
they
probably
don't
even
care
about
your
application
and
that's
because
there's
something
even
more
interesting,
which
is
what's
in
the
dashboard
in
terms
of
your
secrets,
if
you're
keeping
sensitive
credentials
in
your
dashboard
and
using
cronies
secrets,
these
might
be
exposed
in
the
dashboard
and
left
open
to
anyone
who
has
access
to
that,
including
insiders
at
your
company.
A
Someone
can
get
those
those
credentials
and
exploit
them
and
that
typically
leads
to
abuse.
This
is
exactly
what
happened
to
Aviva,
Tesla
and
Weight
Watchers.
In
all
of
these
cases,
the
teams
had
deployed
their
dashboard
on
the
public
web
with
no
password
protection
protection
and
had
privileged
cloud
account
credentials
in
those
dashboards.
In
all
of
these
cases,
hackers
took
these,
or
in
some
cases,
researchers
found
these
logged
in
and
started
mining
cryptocurrency.
A
This,
sadly,
isn't
unusual,
in
fact,
lacework
estimates
that
there
are
over
20,000
public
dashboards,
of
which
300
are
not
password
protected
for
the
next.
In
the
next
up.
Shopify
Shopify
has
a
bug
bounty
program
set
up
for
their
infrastructure,
including
core
technologies
such
as
the
stores
that
they
run,
some
of
which
runs
on
google
kubernetes
engine
it
may
they
got
a
really
terrifying
AG
report
from
researcher
claiming
to
have
root
access
on
all
Shopify
instances.
So
what
happened?
A
While
the
researcher
from
an
arbitrary
Shopify
store,
was
able
to
complete
a
server-side
request,
forgery
using
Google
compute
engine
metadata
from
a
prior
version
of
an
API
and
using
that
they
obtained
the
project
name
an
instance
name
of
where
this
particular
Shopify
cluster
was
running.
With
this
information,
the
researcher
could
submit
a
request
to
the
instance
to
obtain
Q
environment
variables,
including
the
cubelets,
cert
and
private
key.
They
were
able
to
use
that
to
execute
arbitrary
commands
in
the
cloud.
Stur
Shopify
did
a
really
good
job
here.
A
A
It's
for
me,
it's
more
like
an
interesting
study
and
like
what
an
attacker
could
do
and
how
little
people
are
verified
these
images
and
how
much
trust
they
just
put
into
docker
hub
if
I,
if
I
were
an
attacker
I'd,
be
far
more
interested
in
putting
a
backdoor
into
one
of
these
popular
images
and
seeing
what
happens.
One
of
the
really
interesting
things
with
this
particular
article
is
that
people
knew
this
was
happening
and
they
were
complaining
about
it.
A
A
This
means
that
the
best
thing
that
you
can
do
to
protect
your
containers
is
to
patch
them
to
patch
your
base
image
your
packages,
your
application
code,
everything
now
inversely.
What
people
are
really
scared
of,
but
doesn't
actually
happen
in
practice.
Our
container
escapes
a
common
concern
about
containers,
as
they
don't
provide
a
strong
way
to
contain
workloads.
Users
are
hesitant
about
running
multi-tenant
environments,
deploying
to
containers
for
different
workloads
on
the
same
team,
because
they're
worried
about
one
more
code
affecting
another,
those
as
possible
and
test
environments.
A
B
What
other
kind
of
data
does
it
have
access
to?
If
that
containerized
workload
can
be
used
as
a
foothold
to
try
to
compromise
the
operating
system
outside
of
that
container?
What
can
attackers
get
to
right?
And
so,
when
we
think
about
isolation,
we
think
about
these
spreads.
You
know,
while
Maya
mentioned,
we
haven't
really
seen
those
kinds
of
escapes
yet
happening
in
the
wild.
We
do
know
that
it's
a
security
threat
that
researchers
are
looking
at
and
it's
also
one
that
we
take
very
seriously
in
our
threat.
Modeling
at
Google.
B
You
know
in
my
I
gave
example
around
multi-tenancy
and
I
think
there's
a
definitely
a
direction
and
certainly
a
strong
desire
towards
getting
to
a
place
where
you
could
run
multiple
workloads
say,
on
the
same
note,
workloads
from
different
trust
levels
or
workloads
from
different
users.
But
then
we
have
to
also
then
present
that
against
the
challenge
around
containers
not
really
being
designed
as
a
security,
containment
mechanism
from
the
very
start
containers
contain
dependencies.
Containers
are
great
for
deploying
workloads
to
environments
and
moving
them
from
environment
to
environment.
B
Attainers
aren't
exactly
really
the
best
thing
to
think
about
security,
because
the
ultimate
they
just
rely
on
operating
system
security,
which
just
had
its
problems
in
the
past,
okay
and
and
when
we're
thinking
through
these
use
cases
like.
Why
would
you
ever
want
to
think
about
multi-tenant
kind
of
environments
when
we
just
want
to
use?
You
know
one
node
per
customer
or
one
note
per
security
level?
Well,
I
mean
there's,
certainly
a
lot
of
cases
where
you
may
be,
for
example,
intentionally
trying
to
look
at
something.
B
B
B
So
thinking
about
this,
you
know
as
altima,
coming
down
to
a
notion
of
kind
of
trust
and
the
relative
trust
within
the
workloads
and
Trust
is
a
relative
measure
depends
on
where
the
code
is
coming
from.
It
also
depends
on
the
kind
of
data
that
the
workload
is
accessing
and
the
more
sensitive
data
you
process,
the
more
you
need
to
be
able
to
trust
the
code
that
can
access
it.
B
So
when
we
look
at
the
isolation
features
that
exist
in
kubernetes
today,
there's
many
and
they
run
the
whole
gamut
around
different
containment
areas
or
granularity
within
kubernetes
right
from
the
container
of
the
pod
namespaces
around
that
up
to
the
node
level.
Cluster
level
project
level
overall.
B
But
these
mechanisms,
as
they
provide,
may
not
be
strong
enough,
depending
on
the
trust
level
of
your
workload
or
may
require
you
to
fall
back
to
a
coarse
granularity
of
isolation
that
isn't
efficient
to
run,
and
the
other
thing
that
this
diagram
really
says
to
me
is
that
isolation
in
kubernetes
is
very
complex.
There's
a
lot
of
things
to
manage
and
like
with
any
other
security
control,
there's
a
lot
that
users
are
expected
to
get
right
to
configure
and
I.
B
B
So
moving
on
to
looking
at
2018
in
the
last
12
months
or
so
they've
been
actually
some
really
interesting.
Developments
around
isolation
related
to
isolation
for
containers.
12
months
ago
December
at
Q
Khan
in
Austin,
cotta
containers
was
announced,
and
that
was
a
project
that
combined
Intel,
clear
containers
and
Hyper's
run
V.
The
idea
behind
cada
is
that
you
can
build
a
really
lightweight
virtual
machine
and
use
that
to
contain
workloads
and
the
sandbox
environment
has
a
really
stripped
down
guest
operating
system
that
has
less
attack,
surface
and
less
actions
and
less
privilege.
B
B
At
cube,
con
EU,
we
Google
publicly
unveiled
G
visor
Google's
internal
sandboxing
technology,
Google
deploys
over
4
billion
containers
a
week
internally,
and
the
majority
of
untrusted
code
in
those
containers
runs
inside
of
G
visor
at
Google.
What
G
visor
is
is
it
provides
a
strong
isolation.
Boundary
by
intercepting
application
system
calls
at
the
application
layer,
rather
than
sort
of
more
of
traditional
virtual
machines,
which
is
intercepting
things
at
a
lower
machine
level.
B
What
that
means
is
that
we
are
able
to
take
all
this
system
call
interactions,
that
a
workload
would
normally
have
with
an
operating
system
and
encapsulate
and
emulate
that
behavior
and
then
further
sandbox
it
at
Google.
We
have
a
security
guiding
principle
that
we
have
to
strong
security
boundaries
between
untrusted
code
and
the
rest
of
Google
other
workloads
and
data
and
G
visor
helps
us
get
that
moving
on
to
June
of
this
year,
IBM
introduced
nabla
knob
Allah
uses
a
unit
kernel,
which
is
a
library,
OS
kind
of
similar
to
G
Weiser.
B
In
some
ways
it's
a
little
more
streamlined
and
it
still
relies
on
kernel
or
operating
system
isolation.
What
novel
is
just
doing
is
reducing
the
number
of
system
calls
that
need
to
get
passed
through
to
the
underlying
operating
system,
but
doesn't
completely
eliminate
them
different
than
G
Weiser.
That
way,
and
then
last
most
recently
in
November,
Amazon
and
veiled
firecracker,
which
is
a
another
lightweight
virtual
machine
isolation.
Technology
somewhat
similar
to
kata,
but
instead
of
using
the
qmu
virtual
machine
manager
as
kana
does
it
uses
a
custom.
B
B
So,
where
are
we
going
with
all
these
different
technologies?
Right
I
mean
this
is
ultimately
getting
us
to
this
idea
of
actually
getting
native
sandboxing
in
kubernetes
actually
getting
to
containing
containers
from
a
security
point
of
view-
and
I
think
this
is
one
of
the
most
exciting
developments
that
we've
seen
in
2018.
B
Now,
where
are
we
focused
around
the
layer
of
isolation?
Well,
we've
kind
of
settled
in
around
the
pod
being
the
right
granularity
to
think
about
sandboxing
to
think
about
isolation,
and
that
means
that
you
could
have
multiple
containers
per
pod,
but
that
each
of
those
containers
inside
of
the
pod
will
be
inside
the
same
sandbox.
So
there
should
be
relatively
similar
trust
levels
or
users
or
whatever.
B
B
The
initial
implementation
provides
our
runtime
selection
api,
including
a
way
to
select
a
sandbox
technology
that
you
want
to
use
for
an
individual
pod.
It's
implemented
as
a
custom
resource
definition,
ERG
and
defined.
As
part
of
the
pod
specification,
the
runtime
class
gives
users
the
ability
to
select,
as
I
mentioned,
what
runtime
should
be
used
with
each
pod
and
and
why
that's
exposing
that
level
of
detail
is
so
that
users
could
select
sandbox,
runtimes,
seiji
visor
in
some
instances,
cotta
or
nabilah
and
others,
depending
on
their
security.
B
A
Brandon
the
next
topic
suffer
supply
chain
has
also
been
a
hot
topic
on
many
developers
Minds
this
year.
Ultimately,
a
software
supply
chain.
What
we
mean
by
that
is
any
code
that
ends
up
running
a
production
and
the
mechanisms
that
build
package,
and
it
should
be
that
code,
it's
about
making
sure
that
what
you're
running
in
production
is
actually
meant
to
be
there,
that
you
built
it
that
you
verified
it
and
that
you
authorized
it
to
run.
A
This
is
because,
while
developers
and
security
teams
are
awakening
the,
how
do
they
can
use
containers
to
improve
their
security
models?
Hackers
are
also
waking
up
and
saying:
hey
if
I
insert
something
a
little
bit
earlier
in
that
chain,
I
can
get
into
lots
of
companies
workloads
really
easily.
So
what
do
we?
What
do
we
actually
mean
by
a
software
supply
chain
here?
A
Good
security
practices
for
a
container
software
supply
chain
have
a
couple
different
components.
So,
starting
off
with
the
base
image,
you
control
your
base,
image
and
choose
middle
images
that
let
you
limit
the
surface
of
attack,
any
external
images
that
you
want
to
use.
You
can
pull
retag
and
copy
into
your
local
container
image
registry,
the
code.
You
write
your
code,
scan
it
for
potential
security
issues
as
well
as
dependencies,
and,
ideally
you
can
perform
more
complex
security
reviews
and
assessments.
If
you
have
the
time,
builds,
you
build
the
image
yourself.
A
Ideally,
this
is
a
hermetic
reproducible
build
in
the
cutting
edge
here,
something
called
rootless
builds
which
are
builders
where
a
route
is
not
required
to
actually
build
the
image
application
image.
You
scan
the
image
for
vulnerabilities.
This
is
about
decoupling,
your
image
into
the
base
image
and
the
layers
that
sit
on
top
of
it
and
verifying
each
of
those
for
vulnerabilities.
A
You'll,
end
up
scanning,
OS
components,
binaries
jars,
tarballs
everything,
and
you
can
also
scan
your
images
for
misconfigurations
or
if
they
don't
mean
certain
policy
requires
that
you,
but
you
might
have
and
deploy
at
deployment
time.
You
can
enforce
specific
requirements
such
that
your
images
did
in
fact
come
out
of
your
pipeline.
You
can
also
enforce
runtime
requirements
like
that.
A
Your
image
runs
with
set
comp
when
something
changes
like
when
a
new
vulnerability
is
discovered
or
in
a
package
that
you're
using
you
just
go
back
to
this
to
an
earlier
stage
in
the
supply
chain,
and
you
rebuild
and
redeploy
the
image
to
fix
the
issue.
There
have
been
a
lot
of
efforts
in
the
last
year
to
address
these,
and
the
tools
aren't
necessarily
new
this
year
in
2018,
but
the
threat
to
the
supply
chain
is
new,
which
is
why
we've
been
focusing
on
this.
A
So
one
of
the
more
interesting
attacks
we
saw
this
year
was
the
one
we
mentioned
earlier
around
docker
hub
and
we're
likely
to
see
more
of
this
type
of
attack
in
the
future.
There
were
images
on
docker
hub
with
malware
that
that
they
were
meted
with
malware
that
mined
cryptocurrency.
How
could
you
have
avoided
a
similar
attack
in
your
environment?
You
could
not
pull
images
from
public
repositories
directly
instead
adding
them
to
your
local
registry
once
you
verified
them
and
they've
met
certain
needs
that
you
have.
A
You
could
scan
your
images
for
vulnerabilities,
malware
or
other
security
issues
and
only
deploy
the
images
that
you've
scanned,
that
are
in
your
repository
and
that
you
trust
now.
I
can
enforce
lots
of
other
things
like
this
at
deployment
time.
Anything
that's
happened
previously
in
your
supply
chain,
so
that
scan
that
I
just
mentioned,
but
that
it
was
built
by
my
builder,
maybe
that
it
was
built
in
last
30
days
that
QA
test
legal,
whoever
signed
off
on
it,
that
it
was
only
deployed
in
a
particular
geography,
etc.
A
You
can
imagine
all
kinds
of
policies
you
want
that
might
want
on
for
such
appointment
time
so
say
you
only
want
to
focus
on
those
first
items
that
I
only
wanted
to
point
that
have
been
scanning
for
vulnerabilities
and
let's
be
reasonable,
that
don't
have
any
high
unpatched
vulnerabilities.
Note
that
I
didn't
say
no
vulnerabilities,
as
there
might
be
some
for
which
patches
don't
exist,
and
so
your
sets
the
reality
of
you
running
that
workload
and
I
also
specified
high
vulnerabilities,
as
are
the
ones
those
are
the
ones
I'm
most
worried
about.
A
In
my
environment,
Crimmins
has
the
concept
of
an
emission
controller
for
a
plugin,
which
is
a
piece
of
code
that
intercepts
requests
to
the
communities
API
server
and
can
make
admission
decisions.
It
basically
decides
you
shall
or
shall
not
pass.
This
is
pretty
cool
because
it's
a
functionality
that
enables
enforcement
of
what
actually
gets
deployed.
It's
a
single
choke
point
for
everything
that
ends
up
in
your
environment.
So
what
do
I
actually
need
in
my
supply
chain,
people
that
to
enforce
the
requirements?
I
said
earlier.
A
First
I
need
a
way
to
scan
for
vulnerabilities
and
store
information
about
those
scans
and
second
I
need
a
way
to
enforce,
like
with
the
kerbin
I
need
the
mission
controller.
That
images
haven't
been
scanned,
that
images
that
haven't
been
scan
aren't
deployed.
There
are
two
open-source
projects
in
this
space
that
we're
announced
last
December
and
have
gained
popular
in
momentum
now
Griffiths
and
Critias.
So
Griffis
is
a
open
source
project
to
audit
and
govern
the
software
supply
chain,
and
you
can
use
it
to
track
information
like
versioning
and
vulnerabilities.
A
It's
a
little
bit
more,
not
much
more
than
a
key
value,
store,
plus
verification
of
that
store.
Krafayis
stores,
metadata
about
artifacts,
like
images
and
information
about
those
artifacts
like
vulnerabilities,
you
don't
need
to
use
only
one
source
of
metadata
either
you
can
have
multiple
providers,
so
vulnerability.
Scanners,
for
example,
can
each
write
their
own
results?
A
There
are
many
providers,
metadata
providers
like
Jay,
Frog,
Red,
Hat,
core
OS,
IBM
block
duck
twist,
lock
and
aqua
as
your
software
supply
chain
grows
or
involves
you
can
just
add
new
metadata
types
and
new
providers,
then
the
real
benefit
that
you
have
is
that
all
your
metadata
is
in
one
place
about
your
images.
It
lets
you
and
answer
interesting
information
like
if
they're
currently
affected
by
any
recently
discovered
vulnerabilities
criticism,
project
Agra,
face
criticism,
attorneys
and
mission
controller
that
can
block,
builds
based
on
policy.
A
It
has
a
policy
engine
to
test
assertions
before
deploying
into
kubernetes,
for
example,
your
build
provenance
or
the
vulnerability
scans
that
we
just
ran
cluster
admins
manually,
test
images
that
are
verified
and
ok
to
use.
Then
kritis
we'll
only
allow
the
deployment
of
attested
images
kritis
also
integrates
with
it
attestation
metadata
AP
is
so
it
can
speak
the
same
language
as
per
face
so
where
those
put
in
into
our
chain.
Well,
if
you
guessed,
you
know,
one
does
one
and
the
other
one
do
the
oh
one,
you're
good
graffia
stores
metadata
for
image.
A
This
specifically
includes
metadata
for
vulnerabilities
for
scan
results
in
an
attestation
for
those
results,
the
signature
note,
thakura
fayus
isn't
a
vulnerability
scanner.
It's
just
a
consistent
way
of
storing
scan
results
that
you
might
get
from
a
third-party
scanners
and
then
kritis
then
enforces
the
deployment
of
an
image
based
on
the
attestation
information,
ingre
fayus,
there's
a
reference
implementation
of
this
exact
flow
online.
A
An
image
is
added
to
the
registry
plus
anytime.
There's
an
update
to
the
vulnerability
database.
Binary
authorization
is
the
admission
controller
for
crew
nineties
engine.
It
allows
enforcement
of
what's
deployed
in
your
environment
by
signature
verification.
You
can
set
this
up
for
any
kind
of
attestation,
vulnerability
scanner
like
we
talked
about
earlier,
but
also
your
build
pipeline
image
tests,
whatever
you
might
have
taken
together.
A
A
Is
you
know
why
you
should
care
so
much
that
it's
at
running,
containerized
micro
services
with
a
supply
chain
that
you
actually
secured
properly,
leads
to
a
fundamentally
different
security
model
for
containers
than
for
virtual
machines,
so
containers
are
meant
to
be
short-lived
and
frequently
redeployed,
rather
than
pushing
a
single
line
of
code
change,
you're,
rebuilding
and
redeploying
a
whole
container
and
the
whole
image.
If
you
have
other
processes
that
can
take
place
passively
patching
your
OS,
for
example,
you
don't
need
to
have
any
downtime.
A
You
can
actually
just
be
constantly
doing
this
to
your
image.
In
your
image
repository
you
don't
have
to
have
a
Sunday
2:00
a.m.
Nance
window.
You
just
roll
out
that
new
image
the
next
time
you
like,
like
you,
would
roll
out
software
containers
are
also
meant
to
be
immutable,
so
they
don't
change
ones
are
deployed.
A
You
have
content
addressability,
which
means
that
you
know
what's
in
your
environment
so
next
time,
even
if,
when
you
scan
your
image,
it
doesn't
have
any
vulnerabilities
and
then
you
deploy
it,
but
a
new
vulnerabilities
discovered
you
don't
have
to
go
back
and
you'll
have
to
scan
your
live
environment.
You
can
go
scan
your
deployment
history
and
see
if
you're
affected
by
something.
That's
that's
been
newly
discovered
since
you're,
not
changing
some
lines
of
code
but
hold
containers.
A
You
have
that
single
choke
point
for
everything
that
gets
deployed
infrastructure
and
you
can
enforce
governance
so
taking
those
together.
That
tells
a
really
strong
story
as
to
why
you
might
we
don't
have
an
immutable
pipeline?
That's
right!
Immutable
containers
with
a
pipeline
that
you
that
you
have
vulnerability
scanning
in
you
can
enforce
your
policies
in
all
of
that,
together
is
it's
a
very
strong
story
for
security.
B
Right,
thank
you.
So
now
we've
looked
back
at
2018
now.
What
do
we
see
going
forward?
And
that
brings
us
last
set
of
topics
right
so
hardening
in
some
of
our
predictions
for
2019,
so
I'd
say
despite
the
fact
that
we
haven't
seen
many
breakouts
of
containers
this
year,
there
were
certainly
quite
a
few
vulnerabilities
discovered
that
affected
kubernetes
or
really
the
whole
containerize
ecosystem
for
an
infrastructure
service
like
kubernetes.
B
You
have
to
think
about
the
whole
stack
and
patch
issues,
all
the
way
from
really
hardware
up
to
the
kernel
operating
system,
the
VMS
you're
running
kubernetes
itself.
It's
a
lot
to
keep
up
with.
We
had
a
really
exciting
year,
I'd
say
because
we've
had
issues
and
most
all
of
these
layers
this
year,
okay,
and
really
on
top
of
that,
then
you
have
to
think
about
patching
the
applications.
The
workloads
you
run.
B
On
top
of
it,
I
will
note
that
you
know
we
have
a
fantastic
kubernetes
security
team
and
if
ever
you
find
vulnerabilities
or
hear
about
them.
Please
responsibly
report
them
because
that
team
is
responsible
for
developing
the
fix
and
making
the
available
out
to
kubernetes
distributors
and
ultimately
helping
everyone
patch
their
workloads.
B
You
know
and
have
been
again
we're
kind
of
back
to
this
theme
again
around
the
simplest
thing
you
can
do
to
protect
your
cluster
from
attacks
is
to
patch
your
cluster
and
keep
it
up-to-date
with
the
latest
version
of
kubernetes.
If
you're
running
on
the
Google
cloud,
Google's
kubernetes
engine
pushes
out
new
versions
weekly,
which
contains
security
patches,
Jiki
you'll
manage
the
master
for
you,
including
patching
the
master
in
case
of
security
vulnerabilities
for
nodes.
B
If
you're
running
on
container
operas
OS
cause
on
gke,
it
can
patch
them
for
using
node
auto
upgrade,
and
you
can
also
put
me
into
some
windows
around
to
control
when
that
is
done.
So
it's
really
kind
of
a
theme
around
helping
to
automate
and
seamlessly
patch.
Does
the
infrastructure
underneath
the
port
codes,
the
patching
and
on
its
own
is
not
enough
right?
There
are
many
places
where
good
defaults
are
missing
in
kubernetes
and
where
you're
you're
going
to
need
to
do
some
hardening
of
your
own
on
clusters.
B
We
further
published
another
hardening
guide
that
covers
the
the
main
elements
to
dress
they're
kind
of
based
on
the
difference
of
what's
done
by
default
and
what's
not
and
there's
a
link
to
that
at
the
end
of
the
talk
is
well
so
what
you'll
notice,
though,
is
that
previous
list
is,
is
entirely
too
long
right,
there's
just
too
much
that
we
as
a
community
are
asking
average
users
of
kubernetes
manatees
to
take
on
for
themselves
to
get
security
right
and
there's
too
many
chances
to
misconfigure.
There's
too
many
chances
to
get
things
wrong.
B
I
think
this
will
be
a
big
focus
right,
which
is
making
kubernetes
more
opinionated
about
security
and
providing
better
defaults
to
protect
any
and
everyone
using
kubernetes
platform
and
I
think
this
is
something
we
as
security
engineers
need
to
do,
and
you
know
I,
don't
think
we're
doing
enough
there
yet
and
it's
a
responsibility
that
we
take
and
take
pretty
seriously
right,
and
so
that's
going
to
be
in
kind
of
several
major
areas.
That's
going
to
be
around
simpler
configurations
about
getting
templating
about
setting
better
defaults,
removing
permissions
things
like
open
dashboards.
B
Perhaps
getting
a
default
set
comp
profile
into
kubernetes.
You
know,
there's
a
lot
of
wishlist
items,
but
I
think
that
there's
a
patek
for
that
in
2018.
The
other
thing
is
I,
think
we'll
see
a
convergence
in
isolation
technologies.
All
right,
there's
many
options
right
now
go
back
to
that
slide
from
kada
to
G,
visor,
Orton
habla
to
firecracker.
You
got
a
lot
of
options
and
I
think
that
we
might
see
some
convergence
around
that
you
know
and
some
options
may
stand
out
versus
others.
Maybe
some
become
the
leader
around
compatibility.
Some
prefer
forints.
B
So
another
2019
prediction
is
going
to
be
probably
see
more
attacks,
more
container
specific
attacks,
as
more
developers
are
using
containers
and
containers
are
running
more
and
more
interesting
data
is
being
processed
through
them.
Attackers
will
get
more
interested
in
the
containers
and
the
data
themselves
and
the
access
they
provide
beyond
just
the
access
to
a
CPU
to
do
some
cryptocurrency
mining.
So
we
expect
to
see
some
of
the
first
attacks
developed
in
2018,
specifically
targeting
containers
breaking
out
of
containers
going
after
kubernetes.
B
That
way,
I
think
we'll
also
continue
to
see
supply
chain
attacks,
as
Maya
mentioned.
You
know
that's
just
too
open
and
too
ripe
of
an
attack
path
right
now
and
depends
on
too
many
things
to
get
right,
including
you
know,
the
security,
the
internet,
which
we're
like
to
see
I,
think
more
of
those
kinds
of
attacks,
I,
think
I'm.
B
C
There
yeah,
so
it
seems
to
me,
like
there's,
sort
of
an
underlying
assumption,
with
the
isolation
discussion
about
VMs,
being
sort
of
a
impenetrable
isolation,
technology
and
I.
Just
can't
I,
remember
in
my
mind,
back
in
2009-2010,
you
know
hearing
about
privilege,
escalation
issues
with
Xen
and
VMware,
and
then
those
sort
of
cooled
off
so
I.
Just
wonder
if,
if
they're
sort
of
not
just
a
maturation
cycle,
we're
in
yeah.
B
I
mean
I
think
it's
a
fine
question
I
think
really.
The
best
I
would
say
is
that
virtual
machines
are
the
best
isolation
technology
we
have
today.
But
it's
not
it's
not
a
perfect
right.
It's
still
that
there's
a
lot
of
concerns
there
well
I
will
say
is
a
lot
of
investment
in
attention
from
the
security
research
community
has
gone
into
looking
at
VMs
and
VM
escapes
patching
and
fixing
hypervisors
Google.
B
D
B
I
think
well,
it
depends
on
the
the
VM
technology
we're
talking
about
and
how
its
structured
there's
a
layering
question
to
that
conversation
as
well,
which
is
how
many
independent
layers
a
software
getting
exposed
to
untrusted
fork
loads,
and
if
you
had
a
bug
in
one,
how
many
does
that
carry
over
so
I
think
this
is.
The
community
is
obviously
a
conversation
that
we
should
continue
having
and
happy
to
talk
more
about
that
yeah
I.
E
Got
a
question
so,
of
course
isolation,
immutability,
sandboxing,
right
all!
These
are
great
techniques
that
we
get
sort
of
for
free
when
we
go
down
this
route
of
containers
communities.
But
if
you
look
I've
been
studying
breaches
over
the
last
three
or
four
years
in
the
enterprise
they're
not
escape
preaches
right
there,
struts
to
in
a
thirty-year-old
java
application
that,
basically
they
they
hit
a
tomcat
container
and
they
kill
chain
their
way
in
and
and
I
get
how
you
can
mitigate
some
of
that.
E
But
are
we
being
a
little
myopic
here
by
focusing
on
this
kind
of
perimeter,
escape
kernel,
vulnerability
versus
what
most
of
the
banks
are?
The
way
the
real
dangers
are
which
are
like
our
Java
and
those
and
one
more
point,
I
know
I'm
going
on,
but
like
in
your
world
and
a
lot
of
Greenfield
world,
you
have
a
nice
scoping
on
your
dependencies
right
like
you,
don't
go
out
and
use
all
the
libraries
from
Apache
in
in
general,
but
a
bank
like
if
you
look
at
the
dependency
mapping
of
their
core
applications.
E
Ninety
to
ninety-five
percent
of
everything
you
written
is
open-source
and
then
the
dependency
mapping.
You
look
at
the
stretch
to
Jakarta,
breach
that
hit
Equifax
I
mean
the
dependency
list,
is,
will
scare
the
hell
out
of
you
right
and,
and
they
don't
have
tight
control
that
loop.
So
if
I
make
it
sense,
that's
my
kind
of
common
question
for.
B
Sure
yeah
I
think
there's
two
parts
to
that:
I'm
gonna
take
one
part
real,
quick
and
my
I
think
has
a
lot
to
say
on
this
too.
For
that
dependency
side
you
know,
there's
also
the
identification
of
where
that
dependencies
coming
from
what
the
security
stance
of
it
is
and
also
can
we
use
things
like
sandboxing
to
then
take
that
system
that
has
a
lot
of
dependencies
in
sandbox
at
off
such
that.
B
B
B
A
Escapes
are
not
a
thing
that
we're
obviously
seeing
in
the
wild
or
like.
We
know
it
could
happen,
but
that's
right
yet
it's
these
really
boring
attacks
like
we
were
just
saying
it's
like.
Oh,
you
didn't
patch.
You
know
you
have
a
wordpress
with
a
standard
password
like
it's
very,
very
boring.
What
we're
seeing
the
I'm
actually
surprised,
and
now
they
have
a
controversial
opinion.
How
much
people
have
cared
about
isolationist
here,
because
the
only
people
who
should
really
care
are
the
cloud
providers
or
people
running
multi-tenant
workloads
at
scale
right.
A
So,
if
you're
not
running,
if
you're
not
running
your
own
communities,
cluster
and
running
a
multi-tenant
workload
where
you're
taking
arbitrary
container
images,
arbitrary
ml
models,
arbitrary
something
from
your
customers.
This
is
not.
This
shouldn't
be
a
concern
for
you
right,
you're,
you
already
running
in
a
single
time
model.
So
this
isolation
is
it
is.
It
is
obviously
a
very,
very
important
problem.
It's
not
something
that
everyone
needs
to
care
about
which
is
kind
of
interesting
yeah.
C
A
I'm
at
restricting
access
to
who
can
talk
to
talk
to
the
control
plane,
so
on
TCP
the
feature
called
master
authorized
network.
So,
let's
you
restrict
the
set
of
IP
addresses
that
can
talk
to
the
control
plane.
It's
that
I've
seen
customers
build
bastions.
That,
then,
are
they
put
in
front
of
cube
control
that
they
don't
want
someone
to
be
able
to
talk
to.
A
If
you
think
about
how
powerful
cube
control
really
is
at
the
end
of
the
day,
you
might
not
want
the
average
developer
in
your
organization
to
talk
to
it
and
likewise,
if
your,
if
an
average
attacker,
can
somehow
access
it
if
they
touch
some
part
of
your
organization
that
can
be
quite
damaging.
Okay,.
F
So,
to
kind
of
actually
dovetail
on
the
previous
topic,
there
was
an
attack
vector
on
Postgres
earlier
this
year
called
the
scarlet
johansson
attack,
which
you
may
have
remembered
about
which
people
actually
front
door
into
Postgres
and
then
broke
out
of
Postgres
to
get
information
on
the
file
system,
such
as
secrets,
or
things
like
that.
So
I
think
my
question
more
on
the
lines
of.
F
B
Kind
of
the
application,
security
and
sort
of
the
applications,
security,
scans
and
pen
testing
I
think
is
an
area
that
you
know.
We
have
some
interest
in
and
have
been
looking
at,
but
I
think
that's,
probably
an
area.
That's
not
particularly
well
covered.
Yet
right
now,
I
would
say
both
in
the
industry
and
open
source,
so
I
think
that's
definitely
an
area
and
interests
for
sure.
The
other
part
of
that,
too,
is
kind
of
the
questions
around.
B
How
do
you
lock
down
access
from
one
workload
to
another
work
loading
and
what
are
those
interactions
that
they
have
between
themselves?
And
then?
How
do
you
keep
that
from
being
turned
into
a
way
for
an
attacker
to
propagate
from
point
to
point
and
that's
getting
back
to
some
of
the
security
volts
to
templating
the
other
things
where
I
think
we
also
need
to
have
more
attention:
more
investment,
the.
A
Other
concern
I
have
is,
if
you're
running,
two
workloads
in
the
same
cluster,
even
that
have
very
different
security
models.
If
you
look
at
something
like
you
just
mentioned,
logging
like
audit
logs,
are
not
names
based.
So
how
do
you
like,
if
you
can
get
access
to
that,
then
you
can
see
what
the
other
thing
was
doing,
but
you
can't
necessarily
access
the
other
workload
right
and
so
there's
actually
a
really
great
blog
post
by
Tim
all
Claire
that
lays
out
what
isolation
you
get
at
every
level
of
communities.
A
So
that
was
the
image
from
earlier.
What
what
running
in
a
separate
pot
actually
gets
here,
we're
running
in
a
separate.
You
know:
namespace
actually
gets
you
users
ask
me
all
the
time
they're
like
hey
should
I
run
each
or
code
in
its
own
namespace
and
I'm,
like
what's
your
threat
model
like
in
some
cases.
That
makes
a
ton
of
sense
and
in
some
cases
you
actually
want
separate
clusters,
and
in
some
cases
you
want,
you
know
different
projects
or
whatever
it
happens
to
be.
Yes,
so.
G
B
B
G
A
Yeah
you
could
either
scan
scan
was
running
in
production.
Another
method
that
I've
seen
proposed,
although
not
implemented,
was
docker,
was
proposing
doing
something
called
reverse
up
time,
which
is
what,
if
you
just
never
had
anything
that
was
older
than
a
certain
date
running
in
your
environment.
So
if
your
cluster
became
a
month
old,
just
kill
it
and
start
again
from
scratch,
and
the
idea
being
you
know,
everything
in
your
environment
is
less
is
less
than
a
certain
amount
old.
A
B
H
I
have
question
regarding
the
software
supply
chains,
so
far,
I
have
seen
that
it
is
happened
before
the
actually
deploy.
So
what
happened
after
the
company
didn't
deploy
a
single
so
related
to
the
life
scanning,
but
the
other
thing
is
like:
let's
say:
if
it's
one
images,
software
library
that
has
a
CD
it
just
recently
discovers
how
they
is.
This
even
don't
have
any
plans
to
have
this
lifetime
enforcement,
and
we
say:
okay,
we
invalidate
this
image
and
it
will
check
out
our
whole
chance
of
emulating
in
containers
running
on
this
image.
For
example,
you.
A
A
H
But
that's
that
kind
of
scenarios
happen
for
let's
say
developers
or
something
when
we,
when
we
have
actually
control
and
knows
the
library
that
becomes
in
a
lot
of
other
setting
when
we
just
run
a
lot
of
third
parties,
libraries
and
so
on.
We
don't
know
how
to
me
how
much
memory
we
have,
and
it
was
just
like
an
automatically
scan
and
and
database
it
coming
from
aqua
or
coming
on
just
love
and
they
say
hey.
H
B
I
Have
a
question
on
regarding
the
admission
control
you
mention
about
Katie's
I've,
seen
that
there
are
a
lot
of
a
product
in
the
market
like
to
slock
aqua
sake.
They
scan
images.
Is
there
any
push
in
the
industry
to
have
a
common
standard
so
that
we
can
directly
use
those
and
and
use
directly
in
the
admission
control
instead
of
going
multiple
routes?
I
haven't.
A
Seen
the
point
being
yes,
I've
seen
a
lot
of
mission
controllers
across
a
lot
of
different
vendors,
the
two
open
source
ones
that
I
know
of
are
poor,
terrace
and
kritis.
There
might
be
more
apologies
if
I
don't
know
about
them.
The
there
I
haven't
seen
anyone
try
to
push
towards
a
common
standard
there,
the
I
guess
the
standard
is
that
they're
all
kubernetes
and
mission
controllers,
so
that
is
a
standard
configuration
a
standard
way
of
doing
this.
Okay,.
J
K
Maybe
this
is
more
of
just
a
comment
to
add
to
the
discussion
rather
than
a
question,
but
one
of
the
things
that
scares
me
is
the
I
mean
with
the
recent
vulnerability.
We,
you
wouldn't
see
any
logs
for
what
was
actually
done
in
the
cluster
right
or
or
maybe
more
important,
and
an
attack
that
I
think
will
increase
over
the
next
couple
of
years.
K
Is
people
getting
fished
and
then
you
know,
developer
accidentally
exposes
credentials,
and
you
know
now
we
have
some
attacker
keeps
ETL
exacting
into
the
cluster
and
I
mean
there
are
tons
of
vendors
here.
That
will
will
give
you
every
solution
to
to
be
able
to
find
that
and
to
log
even
the
reefs,
the
most
recent
mobility
but
I
I
think
it
would
be
awesome
if,
in
the
community
we
could
also
see
more
abilities
have
better
insight
into
to
what's
going
on
sure
I
guess.
Falco
may,
for
example,
provide
some
of
that
yeah.
L
But
a
lot
of
older
images
are
already
build,
such
that
you
know
they're
they're
Ranas
root.
If
you
pull
the
average
image
off
docker
hub,
most
likely,
it's
gonna
be
running
his
route
and
especially,
if
you
look
at
some
of
the
existing
tool
chains
that
we
have
like
everyone
loves
to
use,
docker
build
by
default
is
gonna,
run
his
route
right.
So
I'm
just
curious,
if
you
guys
have
any
experience
in
terms
of
user
name.
Space
mapping
I.
B
B
J
B
And
I
think
the
other
part
of
that
is
the
namespace
user.
Namespaces
are
a
great
way
to
sort
of
manage
or
reduce
privileges
and
reduce
process
to
process
sort
of
crossover
problems.
It's
not
itself.
At
least
you
know,
consider
it
as
a
strong,
like
security
isolation,
boundary
all
right,
because
it's
still
just
Linux
kernel
boundary,
but
but
certainly
from
managing
permissions
and
managing
privileges.
Yeah.
It's
super
super
useful.
Thank
you.
M
Hi
I'm
I'm
on
a
small
team
and
are
there
tools
available
that
will
make
managing
like
building
my
own
supply
chain
easier.
I
have
a
whole
bunch
of
different
technology
that
we're
using
but
short
of
just
like
me,
monitoring,
github
pages,
to
know
when
they
change
their
docker
file
when
they
make
improvements
to.
Is
there
anything
that
makes
that
easier?
That
workflow.
A
J
A
People
don't
realize
how
bad
the
supply
chain
problem
is
gonna,
get
like
the
NPM
thing
from
a
couple
weeks
ago.
Is
it
for
me:
it's
like
you,
Pearl
Harbor,
like
the
the
canary
in
the
coalmine.
It's
gonna,
get
really
bad,
so
I
think
I
think
we're
likely
to
see
lots
of
stuff,
but
there's
nothing
yet
that
I
know
that's
really
good.
I.
B
A
B
You
know
we
get
the
sandbox
pod
really
there
and
it's
easier
to
take
those
things,
you're,
not
quite
sure
where
they
came
from
and
you
can
fire
that
off
be
fired
off
in
some
sort
of
sandbox
and
then
the
potential,
if
actually
is
bad
or
comes
from
somewhere
strange
or
whatever
the
potential
for
damage
is
now
constrained
or
eliminated.
From
the
rest,
everything
else
you're
running
yeah.
A
I
think
the
best
thing
you
can
do
is
create
the
sounds
again
we're
gonna
say
the
same
thing
be
really
good
at
patch
management,
if
you're
patching
your
things
on
a
regular
basis
and
you're
constantly
rebuilding
redeploying
the
amount
of
time
that
you
be
affected
is
gonna,
be
very
limited
right
because
it's
not
about
you,
know
some
person
making
a
comment
on
some
blog
post
somewhere.
That's
gonna
like
let
you
know
what's
happening,
it's
gonna
be
just
run
the
latest
patch
version
and
the
reality
is
even
though
skin
those
drive-bys
they
were
talking
about
earlier.
A
It's
not
people.
You
know
using
the
the
latest
bone
that
was
dropped
last
week
or
anything
like
that.
It's
people
are
looking
at
Apache
struts.
It's
people
looking
at
things
that
have
been
around
for
years
that
have
been
unpatched.
So
the
advantage
of
moving
to
containers
is
that
you
can
be
constantly
patching
and
you
can
do
that
without
having
significant
downtime,
and
so
you
should.