►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Good Enough for the Finance Industry: Achieving High Security at Scale with Microservices in Kubernetes - Zachary Arnold & Austin Adams, Ygrene Energy Fund (Any Skill Level)
Security is a challenge for most companies. Especially those in periods of rapid growth. It is often taken for granted as we trust the frameworks we use to implement the necessary security protocols for us. However, one only needs to pick up the paper to find out that this simply isn't true. Information security is now a chief concern of many small and mid-size companies as well. We'll show you how we use AWS, Calico, OWASP Dependency Checker, CoreOS’s Clair, and Notary to achieve reliable observable security at the code level, container level, cluster level, and even the AWS account level to maintain the Gold Standard of security (authentication, authorization, and auditing.) We will focus on the security mindset and the specific implementation we have at Ygrene to keep data secure.
About Zach
Zach works currently for Ygrene Energy Fund as a Platform Engineer spearheading the organization's adoption of Kubernetes for production workloads. He works on championing the Microservice movement at Ygrene and is helping to establish information security best practices for the entire PACE industry. He is fascinated by all things Information Security, IOT, Distributed Systems, and his personal hobby is creating Machine Learning models to do all kinds of interesting learning tasks at Ygrene and elsewhere.
About Austin
Software Engineer with 7 years professional experience. People oriented, fast learning and excited to challenge tough problems with modern solutions. Able to lead and follow, has management experience and knows how to see a project through all stages of execution. Loves to have fun, be with family and work in the community. Family man with dreams to have a mini farm and create a sustainable living environment.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Thank
you
guys
so
much
for
joining
this.
The
security
talk.
My
name
is
Zack
Arnold.
This
is
my
colleague
Austin
Adams.
We
both
work
for
a
company
in
the
Bay
Area
called
Y
green
energy
fund,
and
we're
going
to
be
talking
today
about
a
little
bit
about
our
journey
and
security,
specifically
with
cloud
native
tools
and
essentially
attempting
to
demonstrate
that
that
kubernetes
is
actually
ready
for
the
finance
industry.
So
we're
also
going
to
be
talking
a
little
bit
about
achieving
high
high
security
at
scale
with
microcircuits
and
kubernetes.
A
A
I'll
obviously
be
answering
questions
at
the
end,
but
if
you're
not
close
to
a
mic
or
don't
want
to
shout,
you
can
submit
your
question
there
and
we'll
attempt
to
answer
it
at
the
end,
if
there's
time
or
certainly
will
be
available
in
the
corner.
So
what
or,
who
is
why
green,
and
why
do
we
focus
on
security?
Why
green
is
the
word?
Energy
spelled
backwards
in
English,
I
did
not
choose
the
name,
but
it
is
interesting
to
note
I
found
that
out
six
months
into
my
job,
so
that
was
good.
We
are
a
renewable.
A
We're
a
renewable
energy
financing
company
so
basically,
we
operate
under
the
same
or
some
some
laws
that
allow
us
to
apply
the
financing
as
equity,
on
on
a
property
and
essentially
that
gives
us
first
lien
position
against
a
mortgage,
and
that
means
that
we
end
up
interfacing
with
governments,
because
they're
the
only
people
who
are
allowed
to
levy
property
taxes
of
which
why
Green
becomes
a
tax
in
a
way
on
a
property
and
that's
how
you
repay
the
the
loan
or
lien.
So
we
grapple
with
data
integrations
with
other
businesses.
A
The
contractors
who
are
actually
performing
these
improvements
on
properties,
the
the
government's
that
we're
reporting
to
and
and
working
with,
to
offer
this
financing
and,
of
course,
the
property
owners
who
are
giving
us
their
sensitive
information
that
we
want
to
guard
and
protect
in
regards
to
security.
We're
a
lot
like
a
mortgage
banking
provider
in
the
sense
of
that's
the
kind
of
information
that
we
need
to
collect
for
underwriting
loans.
So
you
can
imagine
credit
things
like
that.
A
Are
things
that
we
we
have
to
hold
on
to
and
protect,
and
but
we're
also
in
an
emerging
industry.
So
pace
property
assessed,
clean
energy
pace
has
no
regulations
and
far
as
information
security
is
concerned.
There
is
one
sentence
and
it
says
data
must
be
encrypted
in
transit
and
at
rest,
and
so
for
those
of
you
who
have
dealt
with
any
amount
of
security
at
all,
that's
sort
of
like
the
shrug
emoji
of
security,
it's
sort
of
like
okay,
great
well.
A
What
else
is
there
so
security
matters
to
us
because
we're
a
pace
provider
that
is
aggressively
focused
on
protecting
our
consumers
and
our
property
owners?
It's
extremely
important
to
us
that
we
keep
them
safe
and
that
we
not
only
don't
allow
them
to
make
a
fiscally
irresponsible
decision
by
getting
our
financing,
but
also
that
they
don't
put
their
data
at
unnecessary
risk
when
they
work
with
us
as
a
company.
So
I
guess
then,
if
that's
us,
what's
this
talk
going
to
be
about
what
we're
looking
for
you
guys
to
get
out
of
it?
A
This
is
somewhat
of
a
case
study
for
why
Green
and
and
how
our
security
evolved
as
we
adopted
cloud
native
tools
and
solutions
to
a
lot
of
what
we
did.
We
were
born
in
the
cloud
about
six
years
ago,
so
we
never
really
had
to
worry
about
migrating
legacy
systems,
but
we
did
have
to
deal
with
migrating
legacy
mindsets
when
it
came
to
dealing
with
cloud
native
practices.
A
We
want
to
encourage
you
that
kubernetes
can
and
does
work
for
the
finance
industry.
It
is
safe
and
that
we're
actually
hoping
that
a
lot
of
the
concepts
that
we're
going
to
be
showing
here
and
in
our
implementation
actually
will
be
transferable
to
you
in
your
environment,
regardless
of
what
industry,
you're
in
and
so
to
frame
kind
of
the
next
piece
of
our
talk.
We're
going
to
be
speaking
a
little
bit
on
like
what
we're
calling
the
four
C's
of
where
we
look
at
security
code
containers
clusters
and
the
cloud.
A
Those
are
arbitrary
lines
that
we
drew
just
so
that
we
could
have
areas
of
focus.
It's
obviously
very
fluid
in
there
and
we're
going
to
talk
about
various
pieces
of
all
of
it.
But
first
we're
going
to
be
speaking
about
code
in
containers
and
how
it
pertains
to
the
pipeline,
which
is
really,
where
began,
to
focus
on
security
and
automating.
A
As
much
as
humanly
possible
and
then
we'll
talk
about
the
risk
to
kubernetes
clusters
and
the
tools
that
already
exists,
either
in
kubernetes
core
or
in
the
the
cloud
native
ecosystem
that
help
to
deal
with
these
risks,
so
first
I
guess
to
talk
about
code
and
containers,
is
my
good
friend
and
colleague
Austin
Adams.
So
why
don't
you
welcome
him?
While
he
comes.
B
Thanks,
don't
clap
too
much
because
we
want
to
leave
time
for
questions
at
the
end.
So
can
yes
so
our
journey
with
containers
started
I,
don't
know
two
a
year
year
and
a
half
ago,
two
years
ago
we
were
using
them
in
development,
sort
of
and
I
remember
the
first
time
I
used.
Docker
I
was
completely
confused
or
containers
rather
and
didn't.
Do
it
the
right
way
and
now
I
carry
my
lunch
in
a
docker
container
to
work
everyday.
B
So
for
us,
when
we
started
getting
into
containers,
we
we
realized
that
they're
awesome
in
development.
They
solve
so
many
problems,
but
then
they
add,
maybe
just
as
many
problems
in
production
in
terms
of
more
overhead
for
security
to
manage,
and
you
know
unknown
things
that
can
happen
with
dr.
Tanner
and
a
single
point
of
failure
at
the
docker
daemon.
So
so,
when
we
started
getting
into
containers,
we
identified
kind
of
three
really
important
risks
that
we
wanted
to
to
mitigate
and
some
scenarios
that
go
along
with
those
risks.
B
Like
you
know,
you
push
a
docker
image
and
you
know
the
next
time
you
push.
Oh
the
base
image
updated
because
I
was
using
the
latest
tag
now.
I
have
no
idea
what
version
of
the
OS
I'm
running
and
what
version
of
the
packages
I'm
running
Oh
a
developer
pushed
a
container
from
their
laptop
in
there,
for
whatever
reason
they
have
a
EM
file
or
some
environment
variables
or
some
keys
that
were
in
their
local,
build
context.
B
Also,
you
know,
there's
a
certain
amount
of
docker
so
easy.
So
it's
easy
to
just
kind
of
go
with
a
naive
approach
and
and
end
up
not
knowing
that
you
had
X
package
in
your
base
image,
because
you
just
use
the
regular
basis
that
might
have
come
with
some
random
package
that
nobody
actually
needs
like
image,
magic
or
something
like
that.
I
have
no
idea
or
or
some
obscure
shell.
B
So
what
we
try
it?
What
we're
trying
to
do
at
Y
green
and
it's
very
much
a
journey,
but
what
we've
seen
some
success
and
and
gotten
some
value
out
of
is
creating
a
few
policies
that
match
up
to
those
risks
that
we
identified
and
our
security
iteration
is
to
try
to
automate
the
enforcement
of
those
policies.
So
those
policies
in
name
are-
and
this
is
like
really
small,
so
I'm-
going
to
look
at
the
screen,
which
is
against
the
rules,
but
whatever
a
docker
file
policy,
our
application
and
vulnerability
policies
and
an
identity
policy.
B
So
briefly,
what
that
is,
is
you
know,
I
doctor
file
policies,
everything
about
the
image
as
its
built.
Anything
in
the
docker
file
is
up
for
our
build
server
to
lint
and
to
review
and
to
tell
you
that's
wrong,
a
CVE
policy.
What
sort
of
cv
ease
do
we
even
know
what
to
see
the
the
vulnerabilities
inside
of
our
docker
image
or
our
dependencies
are?
Do
we
even
have
any
clue
what
what
we're
about
to
ship
to
production
and
then,
if
we
do
know
what
are
we
willing
to
accept?
B
What's
too
hard
for
us
to
refactor
or
what
is
our
policy
around?
Do
we
allow
low
low
risk
vulnerabilities
versus
high
vulnerabilities
and
just
getting
some
awareness
there,
an
identity
who's
allowed
to
push
to
production
or
who
is
allowed
to
build
a
container?
Is
it
only
build
servers?
Is
it
developers?
Are
they
allowed
to
build
containers
who
has
to
review
a
container
before
it
goes
out
to
production?
These
are
questions
that
we
need
to
ask,
because
we
have
to
provide
some
sort
of
reporting
and
we
have
to
provide
some
sort
of
auditing
on
everything.
B
That's
happening:
we're
not
processing
transactions
or
anything,
but
we're
dealing
with
a
lot
of
customer
data.
So
if
we
have
a
breach,
we've
got
to
be
able
to
triage
through
and
find
out
who
exactly
caused
that
what
how
it
happened
and
how
we
can
tell
our
the
banks
that
work
with
us
and
the
investors
that
worked
with
us
that
we're
not
gonna
do
this
again,
which
that's
not
happened,
but
you
know
with
docker,
at
least
so
to
kind
of
illustrate
how
we're
automating
those
policies
or
enforcing
those
policies.
B
I'm
gonna
walk
you
through
kind
of
a
typical
security
build
through
our
pipeline.
It's
not
implementation,
specific
I'm,
providing
lots
of
different
tools
and
I'm
not
going
to
go
into
exact
detail
of
how
we
do
it,
partly
because
I
can't
and
other
otherwise,
because
you
know
I
just
want
to
provide
options,
so
it
we
use.
We
use
a
mixture
of
open-source
tools
and
stuff
that
we've
created
that
we
would
love
to
open-source.
B
After
you
know,
we
get
you
know
the
blessing
of
our
company
and
everything
to
do
so
that
are
trying
to
enforce
our
dockerfile
policy
and
our
CVE
policy
and
our
identity
policy.
So
whenever
a
developer
wants
to
make
a
micro
service
a
new
micro
service
or
a
service
at
all,
that
has
a
docker
container
associated
with
it,
they
will,
you
know,
have
to
have
a
build
plan.
B
That's
where
everything
starts,
that's
how
we
do
development
and
they
will
have
to
have
a
docker
file
so
when
their
docker
file
gets
pushed
up
when
they
make
a
commit
to
their
service.
We
use
a
mixture
of
different
docker
file
linters
and
is
that
the
slide
up
there
yeah
and
on
the
screen
and
you
can
download
the
slides,
are
links
to
some
really
good
ones,
and
the
goal
here
at
the
end
of
the
day
is
not
about
the
tools
it's
about
achieving
the
Sippy
standards
for
docker
built.
B
There's
a
section
within
the
docker
benchmark
from
sis.
Pete
am
I,
saying
that
right
that
that
specifies
what
a
docker
file
should
have
and
what
a
docker
file
shouldn't
have-
and
it's
it's
so
incredibly
easy
to
to
lint.
There's
some
tools
as
one
of
these
tools
is
written
in
Haskell.
So
go
you
I,
love
Haskell,
although
I
don't
know
you
guys,
know
I'm
saying
so,
it's
incredibly
easy
to
use
these
tools,
and
so
my
whole
goal
with
you
got
kind
of
this
security
effort.
B
So
that's
how
I
see
it
so
we're
using
these
linters
and
some
homegrown
tools
to
put
policy
around
what
base
images
people
can
use
and
avoiding
best
practice,
screw-ups
like
switching
to
root
and
using
root
as
the
final
user,
not
removing
those
weird
set
good
binaries
and
things
that
have
high
permissions
that
can
do
weird
stuff,
and
this
isn't
about
runtime.
This
is
focused
on
build
time.
Also.
You
know
you
with
with
these
tools,
we're
enforcing
you
know,
hey,
maybe
every
every
part
of
a
certain
date
of
a
classification
has
to
use
Red
Hat.
B
They
can't
use
Debian
or
whatever
it
might
be.
They
have
to
use.
You
know
when
you're
classifying
your
data,
as
you
know,
secure
you
know
top-secret
whatever
it
might
be.
You
know
this
micro
service
might
deal
with
super
secure
data,
so
we
want
it
to
use
a
version
of
Red
Hat.
Maybe
we've
got
a
license
with
them
or
something
like
that,
something
that
is
security
focus.
Where
is
my
graph
on
a--
image?
I?
Don't
I,
don't
really
need
to
have
Red
Hat
or
whatever
your
organization
provides.
B
Maybe
all
of
us
are
doing
this
already,
but
scanning
your
dependencies
of
your
code
I
mean.
Maybe
maybe
people
aren't
doing
this
still,
but
do
it
and
lock
down
your
versions,
you
don't
it's
just
like
docker!
You
don't
want
the
next
bill
to
be
pulling
some
random
version
of
a
dependency
and
then
realize
that
you
have
no
idea.
What's
in
your
container.
Maybe
it's
not
even
unsecured,
or
maybe
it
doesn't
have
a
vulnerability
but
being
able
to
be
100%.
Sure,
what's
inside,
of
your
container
is
actually
really
important.
B
It
helps
with
triage
and
debugging,
and
and
if
something
is
to
happen,
then
you
can
point
back.
Otherwise
you
don't
really
know
what's
going
on
until
you
until
you
check
so
being
100%
sure,
what's
going
out
before
it
goes
out,
is
actually
really
important.
So
again,
the
goal
is
to
look
for
clear
security
issues,
so
we're
using
static
code
analysis
tools,
regardless
of
language,
there's
a
lot
of
them,
a
wasp
azza
and
some
amazing
scanners.
You
can
use
and
and
then
we're
also
scanning
our
dependencies
we're
doing
passive
scanning
just
to
get
alerts.
B
If
there's
a
new
CVE
that
might
relate
to
our
you
know,
code
at
rest,
but
then
also
an
active
scan
of
the
dependencies
as
a
part
of
the
build
process
and
kicking
things
out
if
they
contain
hey.
There's
a
patch
available
for
this
high
security
high.
You
know
this
high
profile
vulnerability
and
you
just
need
to
update
so
we're
we're.
Creating
this
security
culture
through
the
pipeline
by
giving
people
read,
builds
when
they
try
to
push
stuff.
B
B
So
then,
if
everything's
green
on
the
dockerfile
policy
and
the
application
dependency
check,
the
next
step
that
we
do,
is
we
actually
build
the
the
container
you
know
we
have
to
build.
It
turn
it
into
an
to
build
the
docker
file.
Turn
it
into
an
image.
I
think
is
the
right
terminology,
but
we
do
that
in
a
specific
way.
B
So
what
that's
doing
is
that's
giving
us
cryptographic
guarantees
that
somewhat
that
the
author
of
this
base
image
is
actually
who
I
think
it
is
depending
on.
You
know,
lots
of
factors
you
know
if
their
keys
get
compromised.
The
way
that
notary
works
is
it's
very
easy
to
darker
content,
trust
and
know
Tariq
they're
the
same
thing
by
the
way,
it's
very
easy
to
for
them
to
revoke
those
keys
and
and
keep
going,
and
they
can,
you
know,
have
some
transparency
there.
B
But
what
we
want
to
do
is
establish
that
trust
with
those
you
know
very
restricted
set
of
base
images.
Maybe
it's
the
Red
Hat
image
or
it's
the
Amazon
Linux
image
or
the
Debian
image.
Those
base
images
that
you
know
they're
being
signed
by
the
owner
of
a
certain
set
of
private
keys,
that
they
are
keeping
secure.
So
they're,
saying
I,
have
these
keys
I'm
signing
this
metadata
on
the
notary,
server
on
docker
hub
or
whatever
docker
registry
we're
using
we're
pulling
that
metadata
and
actually
validating
that
it's
correct
before
we
before
we
do
anything.
B
B
There's
some
weird
release
things
where
they
might
change
the
checksum
for
a
specific
release,
same
version
number
or
something,
but
we
can
handle
those.
Those
are
like
the
you
know,
the
oddball
cases
they're
not
the
the
norm,
so
we're
trying
to
automate
that.
Maybe
if
you
have
a
tool
that
forces
Dockers
curl
commands
to
check
check
sums,
what
we're
just
gonna
provide
is
a
little
script
that
everyone
has
to
pull.
B
B
All
right
are
we
back
okay,
Thank
You
Zack
for
having
a
thousand
character
password?
Yes,
so
we
squash
so,
for
example,
a
lot
of
times
we
keep
our
dependencies
private.
You
know
we
have
stuff,
we
can't
open
source
right
now
or
whatever,
and
so
we
need
a
s
SH
key,
a
private
key.
So
it's
a
pretty
limited
permissions
key,
but
also
we
don't
want
this
code
getting
into
wrong
hands
and
also
we
don't.
We
don't
want
this
key
in
the
container
at
runtime.
So
this
may
be
straightforward.
B
It's
on
every
blog
out
there
how
to
squash
your
images,
so
I
probably
don't
need
to
tell
how
to
do
it.
If
so,
you
can
google,
but
it's
easy
to
turn
on.
You
basically
have
to
enable
an
experimental
flag
in
the
docker
dr.,
Damon
demon,
Damon,
and
then
you
can
squash
your
images.
What
it
will
do
is
it
basically
just
shoves
everything
into
a
tar
ball
at
the
end.
B
B
So
after
we
build-
and
we
have-
you
know-
ensure
that
there's
some
trust
with
the
base
image
Creator,
which
most
of
the
time
the
base
image
is
that
developers
will
be
using
will
be.
You
know,
DevOps
department
provided
images
because
we
provide
a
lot
of
those
intermediary
images
for
them
will
be
signed
by
us.
So
that's
not
a
big
deal,
but
it's
still
good
to
to
do
because
we
have
our
utilities
in
our
in
our
sea.
I
like
our
Griffin
images.
As
a
other
thing
for
a
CI,
maybe
I
shouldn't
have
told
you.
B
We
use
Griffin
I'm,
not
sure
what
I
can
say
what
I
can't
say
the
next
one
is
we
go
into
some
security
testing,
which
is
my
favorite
part,
and
we
it's
trying
to
automate
the
what
we're,
what
we're,
what
we're?
Okay
with
shipping
in
terms
of
vulnerabilities
that
have
already
been
found.
So
what
we
do
is
we
use
tools
like
core
OS,
Claire
and
Dagda
and
other
image
scanning
systems,
both
passively
and
actively
in
the
pipeline
passively
to
once,
we've
pushed
an
image.
B
You
know
if
there's
a
vulnerability
found
within
the
next
couple
of
days,
we'll
get
an
alert
hey.
These
images
are
vulnerable.
So
then
we
can
take
action
but
then
also
actively
within
the
within
the
within
the
build
to
make
sure
that
this
image
before
we
push
it,
is
not
going
to
have
any
high.
You
know
red
vulnerabilities
or
even
yellow,
depending
on
your
organizational.
You
know
what
what
you're
ok
with
shipping,
then
after
we've
done
that
this
is
my
favorite
part.
B
We
use
the
set
attack
proxy
and
we
pointed
at
the
API
Docs
of
the
micro
service
and
we
see
if
the
Zetas
attack
proxy
can
get
in.
So
if
you
don't
know
what
does
that
attack
proxy?
Is
it's
a
little
not
little?
It's
a
really
cool
penetration,
tester
thing
that
will
use
a
bunch
of
already
known
patterns,
for
you
know
a
bunch
of
different
security
vulnerabilities
and
it
will
actually
attack
some
port
on
your
computer.
B
So
what
we
do
is
we
spin
up
a
set
attack
proxy
and
docker,
and
then
we
spin
up
our
little
service
in
a
little
environment
in
docker
and
we
just
let
the
set
attack
proxy
go
to
work
and
so
again
it's
it's
not
gonna
find
some
zero-day
thing.
Maybe,
but
it's
it's
it's
it's
our
attempt
to
never
ship.
You
know
an
apache
struts
out
of
date.
Now
that
equifax
has
already
been
hacked
by
it
or
whatever
you
know.
B
B
They
created
called
port
eros,
which
is
a
web
hook,
admission
controller
that
allow
that
note
that
works
with
notary,
to
allow
multiple
keys
to
sign
an
image,
and
then
you
can
do
different
types
of
admission
based
on
namespace
and
based
on
keys.
So
what
we
do
is
we?
Obviously
we
sign
our
images
and
for
specific
environments.
We
sign
them
with
specific
signers
right,
so
there's
a
in
in
docker
CLI.
B
It's
this
notion
of
a
signer
which
is
someone
other
than
the
root
key,
the
root
private
key
that
can
sign
an
image,
so
a
trusted
signer
and
then
a
notary.
It's
the
exact
same
thing,
just
a
different
name.
It's
called
a
delegation
key,
so
notary
are
the
doctor.
Cli
uses
the
notary,
NGO
client
to
actually
accomplish
this.
B
So
what
we're
doing
is,
were
you
know,
we're
not
allowing
images
to
go
to
production
that
haven't
been
signed
with
a
specific
key,
but
you
know
developers
can
play
around
in
lower
environments
with
whatever,
but
before
a
container
is
actually
allowed
to
get
into
the
production
cluster.
It
has
to
go
through
some
more
rigorous
examination,
sometimes
some
annual
steps.
Sometimes
just
it's
a
matter
of
just
you
know.
We
just
want
to
be
sure
that
nothing's
going
on
nothing
is
evil
inside
that
container,
so
we're
using
docker
to
our
notary
to
actually
sign
this.
B
But
one
quick
tip
for
notary-
and
this
is
where
I'm
going
to
end-
is,
if
you're
going
to
get
involved
with
notary.
Never
ever
ever
ever
ever
use
your
route
or
target
keys.
So,
and
you
should
learn
you.
Can
you
can
look
at
the
docs
to
learn
about
how
notary
works,
but
basically
you're
issued
thousand
different
private
keys,
two
of
which
are
I,
know
it's
kind
of
crazy,
we're
working
on
that
in
the
notary
project
to
make
that
better
and
easier
to
use.
B
But
you
get
a
root
key
in
a
target
key
and
those
can
remain
offline.
You
don't
need
them
to
actually
sign
images.
You
need
them
just
to
create
these
signer
keys.
These
trusted
keys.
So
if
any
of
these
delegation
keys
get
compromised
because
you
have
kept
these
root
Keys
offline
in
like
a
flash
drive
or
in
a
security
device
or
something
you
can
actually
revoke
privileges
on
those
keys,
those
keys
are,
you
know,
derived
from
those
other
keys,
but
it
makes
it
easy
to
recover
from
a
breach.
B
So
you
can
be
pretty
sure
that
even
if
you
you
do
get
a
build,
server
gets
hacked
and
they
get
the
key
and
they're
signing
images
and
pushing
them.
You
can
revoke
those
and,
and
then
you
don't
even
have
to
delete
them
out
of
your
registry,
your
kubernetes
cluster,
when
it
pulls
that
image
or
your
admission
controller.
When
it
pulls
that
will
recognize.
Oh,
this
key
ID
is
not
within
the
signers
list
on
the
notary
server,
so
I'm,
not
gonna.
Let
this
image
go
through.
So
that's
what
we're
doing
in
our
build
pipeline.
B
I
hope
that's
useful,
maybe
inspirational
at
least,
and
so
obviously
the
container
has
to
go
somewhere
so
to
finish
off
the
security
that
we're
doing
that.
Why
green
zach
is
gonna
talk
about
how
we're
doing
it
in
the
cluster
and
the
cloud
actually
running.
These
containers
we've
spent
so
much
time
making
that
sure
that
they're
secure
thanks.
A
Austin
so
I
guess:
let's
now
talk
about
the
last
two
seeds
which
are
the
cluster
in
the
cloud
it
is,
this
is
probably
going
to
make
the
most
sense.
I
was
going
to
ask
actually
are
there
any
sis,
offs,
admins
or
DevOps
folks
in
the
in
the
car
dev
check
offs
types?
Okay,
a
few
of
you
nice.
You
guys
escaped
wonderful,
so
the
the
whole
notion
of
running
software
in
production
is
obviously
nothing
new
to
any
industry.
A
Stability
is
extremely
important
and
kubernetes
does
a
lot
of
that
out
of
the
box,
but
as
far
as
running
software
securely
in
production,
that
is,
it's
actually
been
a
growing
field
and
growing
every
single
day
I
mean
the
most
hotly
coveted
position
in
the
c-suite.
Now
is
the
sea,
so
the
chief
information,
security
officer,
so
I
guess,
let's
talk
a
little
bit
about
issues
and
kubernetes
for
the
risk-adverse
or
the
the
people
who
are
going
to
be
running
your
clusters
in
production,
the
software.
A
So
it's
important
to
note
that
the
locus
of
control
for
you
as
a
software
operator,
someone
who
deploys
code
onto
the
cloud
that
is
no
longer
you
that
is
kubernetes
the
kubernetes-
will
be
monitoring.
Restarting
pods,
deploying
scaling
handling
a
lot
of
those
concerns
for
you.
So
there's
a
lot
less
control
over
that
final
resulting
environment,
which
means
you
really
have
to
nail
what
we
spent
most
of
our
time
on,
which
is
the
the
docker
image
that
eventually
gets
deployed
onto
the
cluster
security,
really
starts
in
the
build
pipeline
for
us.
A
So
if
you
understand
that
the
locus
of
control
moves
from
you
to
the
cluster
and
the
are
the
way
that
we
think
of
the
kubernetes
cluster
is
essentially
it's
a
benevolent
botnet,
it's
the
command
and
control
servers
or
the
API
masters
and
the
couplets
are
the
nodes
which
is
kind
of
funny.
When
you
think
about
it
and
then
all
of
a
sudden
you're,
like
oh
I,
hope
I
never
get
compromised
because
it's
a
botnet,
so
it
could
become
a
malevolent
botnet
and
that's
not
obviously
something
you
want.
A
So
it's
guess
it's
important
to
note
that
this
is
open
source
software,
which
means
it's
generally
extremely
well
reviewed
by
a
lot
of
brilliant
people.
Some
of
you
here
are
even
contributors.
Thank
you.
So
very
much
you've
helped
us
to
do
well
as
a
business,
but
that
does
not
make
it
flawless.
They
mean
just
two
months
ago
with
the
host
Pat
host
path.
A
Mapping
issue
that
was
released
as
a
CV
in
kubernetes
was
one
of
the
more
serious
that
happened
and
there
was
a
patch
rolled
out
in
about
24
hours,
but
that
doesn't
mean
that
every
single
cluster
that
had
that
vulnerability
also
had
that
patch.
So
it's
extremely
important
that,
as
a
cluster
operator,
we
at
Y
green
we're
subscribed
to
every
CVE
mailing
list
that
we
can
the
kubernetes
announced.
A
Jose
is
a
group
that
you
they'll,
let
you
know
if
there's
something
coming
in
as
a
security,
vulnerability,
yeah,
it's
a
Google
group,
so
kubernetes
announced
it's
important
to
be
aware
of
that,
because,
if
you're
going
to
run
software
securely
in
production,
but
your
underlying
for
structure
is
absolutely
hosed,
then
it
really
doesn't
actually
matter
how
secure
your
container
is,
because
your
environment
isn't
also
important
to
note
that
your
attack
service
essentially
doubles
when
you're
getting
into
kubernetes,
and
that's
one
of
the
considerations
that
we
had
to
make
it.
Why
Green?
A
Were
we
comfortable
with
that?
Because
traditionally,
when
you
look
at
vulnerabilities
and
threat,
modeling
you're,
like
okay
I,
think
that
a
vulnerability
could
come
from
the
the
hypervisor
or
the
VM.
It
could
come
from
the
network
or
it
could
come
from
the
software
I
deploy
onto
it.
Well
with
kubernetes.
You
have
not
just
that,
but
you
have
also
a
mini
hypervisor
called
the
doctor
daemon,
a
mini
VM,
which
is
called
a
docker
container,
a
container
networking
interface
and,
of
course,
obviously
the
software
that
orchestrates.
All
of
that.
A
There's
there's
current
discussions
going
on
between
all
the
SIG's
of
like
who's
really
responsible
for
determining
the
overall
security
of
this
this
environment.
So
if
those
are
the
concerns,
then
here
are
some
mitigations.
A
lot
of
these
will
be
extremely
familiar
to
those
of
you
who
runs
secure
software
in
production.
Kubernetes
basically
has
abstractions
for
everything
that
that
you
would
traditionally
run
in
a
cloud.
Here
is
a
quick
list:
it's
by
no
means
exhaustive,
but
I'm
going
to
run
through
it
really
quickly.
So
we
can
get
some
questions
we're
running
a
time.
A
Our
back
role-based
access
control,
it's
right
out,
happens
right
after
authorized,
eight
are
authentication
and
the
api
server
it
works.
Just
like
role-based
access
control.
You
can
set
up.
All
of
your
principles,
objects
verbs,
just
ensuring
that
whoever
is
talking
to
your
kubernetes
cluster
is
allowed
to
do
what
it
is
that
they
just
asked
for.
If
they're
trying
to
delete
your
production
deployment
of
some
extremely
critical
services,
do
they
actually
have
the
permission
to
do
that?
Auerbach
will
help
cover
that
API
server,
admission
controllers-
these
are
beautiful,
learn
these
love!
These
build
your
own.
A
If
you
want
to
API
server,
admission
controllers
happen
right
after
the
authentication
and
authorization
step
of
kubernetes.
It's
essentially
one
more
place
where
the
API
server
can
check
to
say.
Is
this
object,
okay
to
go
into
the
cluster
or
does
something
need
to
change?
So
it
makes
sense.
There's
a
well
published
set
of
defaults,
validating
admission
controller
basically
says
yes,
no,
this
is
allowed
to
go
in
an
example
would
be.
Is
this
pod
trying
to
run
docker
privilege
mode?
A
No,
don't
let
it
in
a
mutating
webhook,
which
was
our
friends
over
from
IBM
and
porty
Eris,
which
was
actually
changed.
The
object
to
match
the
specification
that
you
have
defined
before
it
goes
into
the
cluster
there's
debate
on
whether
or
not
that's
actually
what
you
want
to
do
if
you
should
just
outright
reject
an
API
server
request,
because
technically
it
changes
before
it
goes
in
the
cluster.
A
So
it's
not
exactly
what
you
sent,
but
actually
it
works
out
in
the
case
that
IBM
mentioned
audit
logging
is
new
in
1.9
its
beta
right
now
in
1.10,
it's
terrific.
It
is
as
good
as
you
make
it.
So
if
you
have
a
coupon
ease
cluster
and
you're,
just
passing
around
your
coop
config
file
to
everybody
and
then
it's
that
essentially
makes
everybody
god
mode
on
the
cluster.
A
That
is
probably
not
going
to
be
super
safer
or
helpful,
and
if
you're
doing
issue
diagnosis
or
triage,
then
every
single
request
from
the
API
server
point
of
view
looks
the
same.
So
that's
not
very
helpful.
It's
good
to
partner
that,
with
an
IDP
like
hep,
Tio's,
AWS
Authenticator
for
running
an
AWS
or
core
OS
is
Dex
and
we'll
talk
about
in
a
moment,
network
security
policies,
pod
security
policies.
Essentially
they
give
you
the
control
around
your
environment
that
we
used
to
have
a
things
like
AWS
security
groups
or
things
like
setting
up
SELinux
profiles.
A
I
won't
go
super
deep
for
them
because,
like
I
said
we're
only
a
time
and
then,
if
that
isn't
enough
control
for
you
scheduling,
taints
and
toleration
z',
these
are
actually
kind
of
cool.
Essentially,
you
can
make
sure
that
sensitive
workloads
and
non
sensitive
workloads
are
not
Co
scheduled.
So
even
if
you
have
a
node
compromised,
there
is
still
separation.
So
you
wouldn't
want
a
Drupal
content
management
website
running
next
to
something
that
pulls
credit.
A
So
then
you
can
schedule
those
so
that
they
take
nor
tolerate,
and
you
can
do
pod
affinity
or
anti
affinity
to
make
sure
that
they're
not
actually
running
on
the
same
node
inside
of
your
cluster.
So
then
you
don't
end
up
having
a
problem
where,
if
Drupal,
which
they
had
a
zero
day
a
few
days
ago,
gets
hacked
and
someone
runs
a
shell
code
attack
and
somehow
they
managed
to
escape
the
dr.
A
Damon
and
they're
running
in
route
inside
of
your
credit
polling
service
cool,
so
some
external
tools,
aqua
scanners,
COO
bench
project,
Thank,
You,
aqua
scanner-
if
you're
here
they
do
see,
is
for
for
the
kubernetes
environment.
It's
it's!
It's
pretty
good.
You
basically
can
continuously
check
your
conformance
with
the
CIS
standards
that
are
published
for
kubernetes
right
now.
Obviously,
I
am
and
then
communication
in
between
services.
Those
can
be
secured
with
a
service
mesh
like
sto
or
linker
D
or
conduit
or
one
of
those,
and
that
is
absolutely
incredible.
A
A
We
were
in
AWS
before
eks
is
the
thing:
it's
still
not
a
thing
AWS
if
you're
listening,
please,
please
make
it
a
thing,
but
if
you
can't,
then
you
can
use
a
tool
like
cops,
it's
great
because
you
can
version
your
cluster.
It
has
an
automated
update
and
rollback
strategy
allows
you
to
configure
everything
but
you're
kind
of
on
your
own
for
issue,
diagnosis,
and
then
it
has
a
high
management
overhead
I
mean
there's
people
who
have
full
time
jobs
now
in
companies
to
make
sure
that
their
clusters
are
alive
with
that.