►
Description
Dance Madly on the Lip of a Volcano with Security Release Processes [I] - Jess Frazelle, Google & Brandon Philips, CoreOS
This talk will cover how we designed an awesome security release process for Kubernetes and all it’s sub-projects.
Open source projects strive to be transparent in everything they do, but when it comes to fixing security patches they need to find the right balance of “open” and “responsible.” This means vulnerabilities should be reported in a safe way as well as patches tested and reviewed with a limited audience. The companies that rely on Kubernetes should have time to patch their systems before a public announcement.
Various sets of infrastructure and collaboration are needed to make this a reality. The design we used could also be applied to other projects and even internally in your company.
Join us to learn about the Kubernetes Security Release process and how we went from no infrastructure in 2016 to great infrastructure backed by an awesome team in 2017.
About Jess Frazelle
Jess Frazelle is a Software Engineer at Google. She has been a maintainer of Docker, contributor to RunC, Kubernetes and Golang as well as other projects. She loves all things involving Linux namespaces and cgroups and is probably most well known for running desktop applications in containers. She maintained the AppArmor, Seccomp, and SELinux bits in Docker and is quite familiar with locking down containers.
About Brandon Philips
Brandon Philips is helping to build modern Linux server infrastructure at CoreOS as CTO. Prior to CoreOS, he worked at Rackspace hacking on cloud monitoring and was a Linux kernel developer at SUSE. As a graduate of Oregon State's Open Source Lab he is passionate about open source technologies.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Why?
Why
is
security
important
at
all?
Why
is
that
even
part
of
our
discussion
I
like
to
put
it
this
way,
we
built
the
world's
most
secure
computer
right.
We
did
that
a
long
time
ago,
they're
not
Network,
connected
physical
input,
visual
output,
easy,
but
actually
today's
systems
are
all
network
connected,
and
so
security
is
all
about.
Ensuring
that
whatever
is
connected
to
the
network,
doesn't
have
any
bugs
in
it,
and
if
something
does
happen,
where
a
bug
is
found
on
the
network
expose
above
you
the
network,
you
have
to
do
something
about
it.
A
You
have
to
do
something
about
it
very
rapidly,
because
this
is
essentially
the
feeling
that
we
all
get
when
a
zero-day
comes
out
of
a
system
that
we're
maintaining.
This
is
Apple
engineers
finding
out
about
an
iPhone
vulnerability,
but
I
think
we've
all
been
in
an
empathetic
position
before,
and
this
is
where
the
name
of
the
talk
comes
from.
A
A
If
you
do
the
math,
it
doesn't
look
good
for
us
and
we
have
to
be
very
leveraged
with
what
we
do
so
also
some
people
say:
well,
maybe
we'll
catch
up,
we'll
just
train
more
software
engineers
up
I,
don't
think
the
numbers
look
good
here
either
about
a
quarter
of
a
billion
people
are
added
to
the
internet
every
year
if
this
trend
continues.
So
why
is
it
important
it's
because
everyone's
data
is
moving
on
to
servers.
This
is
the
dominant
paradigm
of
today.
A
I
know
a
lot
of
us
as
software
engineers
as
people
who
care
about
privacy,
etc.
Don't
like
this
paradigm
philosophically,
but
it
has
won
in
the
market
and,
to
a
large
degree,
we're
responsible
for
the
systems
that
put
everyone's
documents,
commerce
and
communications
onto
a
server
somewhere
and
so
estimated
worldwide.
There's
about
a
hundred
million
servers,
there's
30
million
of
us.
So
that's
three
per
person.
B
B
But
we
can
let
vendors
know
and
vendors
won't
break
embargo,
which
we
actually
had
a
problem
with
one
vendor
breaking
embargo
when
I
was
at
our
docker
and
we
had
to
do
the
release
like
right
away
because
of
it,
and
so
it's
hard
to
kind
of
keep
this
balance.
But
you
can
have
both.
You
know,
I'd
make
Iron
Man
here
happy,
so
there's
a
bunch
of
like
different
rabbit
holes
that
you
can
go
down.
Gpg
is
one
of
them,
like
some
security
researchers
want
to
send
you
encrypted.
B
You
know
messages
about
the
vulnerability
and
the
details
of
it.
Some
don't
care,
and
then
it
leaves
down
you
know:
do
you
want
the
entire
security
team
to
have
to
share
one
GB
GT,
which
is
a
little
bit
scary?
Do
you
want
to
post?
You
know
your
individual
GPG
teams
and
then
they
have
to
email
everyone.
So
there's
kind
of
like
a
nice
balance
here
too,
that
you
like
can
decide
on
as
far
as
encryption
of
the
process
as
well,
and
then
you
have
to
try
making
everyone
happy
which
this
is
well.
B
My
favorite
gets
four
opinions
on
things,
so
you
have
the
researchers,
you
have
the
community
and
you
have
the
vendors
and
all
the
sides
to
kind
of
have
their
own
priorities
of
like
what
they
want
from
this
process.
You
know
so
the
user
is
like
you,
obviously
don't
want
bugs
in
your
code.
You
don't
want
malicious
people
on
your
servers
because
in
Otay
never
gothics.
B
You
also
want
to
be
able
to
upgrade
without
like
this
thing,
taking
down
your
entire
cluster
and
everything
with
it,
and
then
also
you
want
to
be
able
to
know
when
security
fixes
are
there,
so
that
you
can
upgrade,
and
you
can
act
on
them
accordingly,
like
based
off
how
severe
it
is.
If
you
need
to
upgrade,
then
you
need
to
know
about
it
ahead
of
time,
because
you
have
literally
so
many
servers
exposed
that
you
don't
know
what
to
do
with
it.
B
So
then
you
have
also
the
researcher
or
you
know,
original
reporters
expectation
which
I
tried
to
talk
to
like
a
few
of
my
friends,
and
they
were
like.
We
just
want
to
know,
what's
happening
because
apparently,
like
most
vendors,
don't
tell
you
anything
like
you'll
try
to
report
it
and
they'll
just
be
like
either
we
don't
err
or
it
just
goes
into
some
like
null
void
somewhere,
and
you
never
hear
about
it
again
and
that's
when
they
start
doing
like
these
public
disclosures
without
them
fixing
the
bug,
which
is
a
problem.
B
So
they're
like
just
like
a
kit
when
you
get
it
then
give
us
updates
randomly
and,
like
maybe
you
know,
they
aren't
emailing
us
every
single
month
like
what's
happening
with
it.
We're
actually
giving
them
updates,
like
as
things
happen
and
then
there's
the
project's
expectations,
which
is
us
which
is
cool.
So
obviously
we
want
bugs
to
be
fixed
quickly
like
we
want
embargo
to
be
kept
secret.
B
So
there's
also
a
bunch
of
problems
with
getting
people
to
upgrade.
You
have
to
make
sure
people
know
whether
this
is
over
mailing
list.
Twitter,
you
know,
carrier
pigeon,
so
there's
like
a
bunch
of
different
factors
involved
here
and
kind
of
the
project
itself
has
to
establish
a
pattern
that
people
get
used
to,
and
then
you
have
to
continue
that
pattern
so
that
it
is
consistent
and
people
can
consistent
consistently
like
know
that
they
need
to
upgrade,
and
then
there's
also
what
you
have
to
do
in
terms
of
handling
it.
B
A
So
I
wanted
to
play
this
game
really
quick,
I'm,
going
to
call
them
people
to
raise
your
hands.
I'm
gonna
ask
for
each
of
these.
What
the
piece
of
software
was
that
was
vulnerable
and
then
the
name
of
the
vulnerability,
and
so
we'll
go
from
left
to
right.
So
so
anyone
so
I'm
gonna
answer
the
first
one.
A
A
A
B
The
bugs
that
effect
anything
of
mine
so
there's
also
a
bunch
of
problems
with
how
bugs
themselves
get
reported.
Sometimes
it's
like
a
security
researcher
who
actually
like
knows
this
process
and
kind
of
like
notes
them
beans
and
so
they'll
be
like
you
know,
I
found
a
bug,
should
I
be
evil
or
should
I,
let
upshore
know
or
they
like
Google.
You
know
our
security,
disclosure
policy
or
they'll
just
straight
up
email,
security
at
which
is
where
I
forward,
like
all
my
Apple
phishing
emails,
to
which
I
realized
was
actually
a
thing.
B
I
asked
them
one
one
day,
so
they
can
look
it
up,
but
the
problem
like
lies
when
people
don't
realize
kind
of
what
the
bug
severity
is.
So
they
think
it's
just
a
bug,
but
the
underlying
issue
could
cause
all
these
like
problems
with
access
to
data
and
stuff,
like
that,
so
they'll
file
an
issue
on
the
repo,
and
this
is
how
mistakes
happen,
and
it
you
need
someone
almost
to
recognize
right
away,
that
that
is
a
problem,
and
we
have
this
happen
several
times
on
docker
one
of
the
times.
B
I
actually
has
it
call
a
friend
at
github
and
be
like
delete
this
issue,
which
actually
then
become
super
sketch.
So
there's
kind
of
like
also
a
balance
here,
where
you
know
whether
it's
you
know
changing
the
issue.
Contents
and
letting
the
original
reporter
know
the
severity
and
why
you
changed
it
first
or
you
know
just
kind
of
slowly
shoving
it
under
the
rug
until
you
can
fix
it
and
make
it
like
a
thing
that
is
then
publicly
disclosed.
A
So
one
of
the
things
before
like
just
designing
a
process
and
vacuum,
we
wanted
to
go
through
some
of
the
communities
that
we
respect
like
just
spend
a
lot
of
time
in
the
doctor.
Community
and
I've
spent
some
time
in
other
communities.
So
just
going
to
give
you
a
survey
of
some
of
the
lessons
that
we
included
in
the
community
security
process
that
we
learn
from
others
and
attribute
them.
So
there's
lunch
of
just
common
best
practices
that
everyone
should
really
follow
in
these
security
processes
on
the
first
is.
A
A
Having
internal
documentation
about
the
process
so
that
it's
clear
who
actually
owns
something
once
it
comes
to
the
list
having
a
security
leveling
system
so
that
you
can
say
at
least
through
some
rough
metric,
how
severe
this
is
and
talk
about
it
in
an
objective
way.
Everyone
will
naturally
say
that
their
bug
is
the
worst
thing
ever
found
ever
and
just
having
some
communication
around.
That
is
super
important
and
then
requesting
CVS
when
it's
appropriate.
A
B
With
stalker
it
was
really
nice.
We
tried
to
make
sure
the
entire
process
was
documented
in
the
open
source
repo
and
that
almost
anyone
like
who
kind
of
just
stepped
in
one
day
they
could
be
like
I,
can
right
now
this
day
to
a
security
release,
which
is
important
if,
like
the
one
person
goes
out
of
town,
and
then
there
is
like
tooling
for
everything
all
the
scripts
to
build
and
release
stalker
can
be
run.
You
know
anywhere
at
any
time
by
anyone.
A
So
this
is
something
that
I
learned
from
Greg
Hartmann
and
the
Linux
kernel.
One
thing
that
often
you'll
get
into
binds
with
security
researchers
will
they'll
ask
to
essentially
negotiate
the
timelines,
they'll
disclose
it
to
you,
say
they'd
like
to
have
it
fixed,
but
then
they're
going
to
be
giving
a
sweet
talk
at
TCC
or
something
eight
months
from
now,
and
it's
like
you
can't,
you
can't
actually
allow
a
secured
known
security
vulnerability
to
sit
around
until
the
researchers
paper
comes
out
or
it's
convenient
for
them.
So
this
is
a.
A
This
is
something
that
the
Linux
kernel
states
buy
and
they
set
expectations
up
front.
If
you
disclose
it
to
us,
we
will
process
it
within
the
order
of
weeks
so
less
than
a
month
and
get
it
in
front
of
users
as
a
six
another
one
that
I
thought
was
really
interesting.
I
hope
to
do
in
the
future
with
kubernetes
is
no
Japs
has
a
core
security
team
and
in
a
community
security
team
that
actually
helps
all
the
ancillary
projects
that
make
no
js'
successful,
have
a
good
security
process.
A
This
is
really
really
cool,
because
inside
of
kubernetes
is
not
just
kubernetes
core
that
people
are
running
they're,
going
to
be
running
networking
systems
and
storage
systems
and
a
bunch
of
other
containers
that
don't
necessarily
come
out
of
the
kubernetes
git
repo
and
something
that
we
should
definitely
start
to.
Think
about
as
the
sort
of
franchise
of
kubernetes
grows
and
grows.
B
So
with
go,
what
I
really
liked
was
Bill
email.
You
know
this
like
go
announce
mailing
list,
which
is
basically
the
equivalent
of
our
announced
and
they'll
say
you
know
we
have
a
security
vulnerability
and
in
a
week-
or
so
you
know
we're
going
to
release
this
dot
version
with
it,
so
that
we
ourselves
could
prepare
our
own
projects
to
upgrade,
and
we
kind
of
like
knew
like
impending
doom
is
coming
kind
of
which
is
nice,
and
then
they
have
this
on
their
security
release
page
they
have
like
a
super
small
team.
B
It's
really
just
like
Android
ran
and
Adam
Langley
and
their
email
addresses
are
there
with
their
GPG
key.
So
you
can,
you
know,
just
go
straight
to
them
depending
on
the
severity
or
you
know,
they'll
see
that
faster
than
with
any
sort
of
mailing
list,
and
it's
super
nice
that
you
have.
You
know
exactly
who's
on
there
to
handle
your
issue
right
away.
Yeah.
A
This
is
something
that
I've
found
for
a
lot
of
like
disclosure
offices,
weather,
like
code
code
of
conduct,
disclosures
or
security
disclosures
listing
a
listing
explicitly.
The
people
who
are
behind
that
list
is
really
important
because
you
may
want
to
track
them
down
on
Twitter.
You
may
want
to
have
a
face-to-face
conversation
with
them,
etc,
and
so
I
think
it's
just
in
general.
Mailing
lists
that
are
like
this
is
a
team
should
list
who's
on
the
team
cool.
A
A
Okay,
so
kubernetes
normal
release
process,
we
actually
have
a
really
really
sophisticated
release
process
that
people
aren't
familiar
with
it
for
an
open-source
project,
it's
extremely
mature.
So
at
any
given
point,
we
have
three
minor
releases
of
kubernetes
flying
at
the
same
time
and
there's
release
managers
for
each
one
of
these.
So
your
release
manager,
four
one
five,
one
four
and
there's
a
release
manager
for
each
of
these.
A
So
we
have
three
minor
releases
that
are
getting
bug,
fixes
and
security
patches,
and
that
gives
users
roughly
about
9
to
12
months
of
security
patches
that
they
like
deployed
one
five
they'll
have
about
nine
to
twelve
months
before
the
security
patches
stop
going
to
one
five
and
they
need
to
move
to
1/6,
and
then
this
this
whole
process
is
maintained
by
a
set
of
volunteers
that
actually
build
the
release.
Take
the
cherry-pick
review
the
thing
and
run
the
process
yeah.
So
it's
all
about
nine
months
and
then
for
each
of
the
minor
releases.
A
We
actually
have
this
whole
process,
which
Dan
Gillespie
from
core
OS
just
got
one
six
out
the
door
yesterday,
but
it's
actually
a
big
team
effort
of
a
big
schedule
with
these
quarterly
quarterly
releases
of
kubernetes,
where
we
have
a
set
of
alphas
and
betas
and
RCS
and
then
finally
a
final
release,
and
it
takes
about
six
six
weeks
or
so
to
run
through
the
full
process
and
the
teams.
We
have
very
sophisticated
set
of
people
who
make
these
releases
possible.
A
It
goes
all
the
way
from
the
release
manager
who
is
overseeing
and
is
ultimately
accountable
for
on-time
successful
release,
but
a
ton
of
people
make
these
things
possible
and
it's
a
set
of
volunteers
that
do
all
this
work.
So
you
have
people
who
are
wrangling,
the
features
that
we
know
actually
within
the
release.
We
have
people
who
are
writing
any
of
the
documentation
for
migrations
or
deprecations
or
things
that
would
actually
affect
users
moving
from
one
release
to
the
other.
We
have
people
accountable
for
testing
to
ensure
that
all
the
automated
tests
are
passing.
A
There's
very
brave
souls
that
do
manual
testing
to
ensure
that
actually,
the
product
works
as
expected,
not
just
that
the
tests
are
all
green,
which
is
pretty
critical
because
at
the
end
of
the
day
a
human
has
to
use
it,
and
so
on
so
forth
through
the
whole
through
the
whole
release
process,
and
so
there's
about
eight
folks
that
made
this
one
six
release
possible
and
it's
it's
dedicated
work.
It
takes
about
six
to
eight
weeks
of
just
hard
work
of
all
you're
doing
is
focusing
on
release
success.
A
And
so
how
we
actually
get
to
successful
release
I
like
this
diagram.
This
is
a
diagram
that
you
see
on
kubernetes
IO
on
the
front
page
I
like
it,
because
it
makes
kubernetes
look
really
complex,
there's
like
a
lot
of
stuff
in
there,
and
so
the
importance
that
I
wanted
to
highlight
here
is
that
there's
actually
a
lot
of
stuff
in
kubernetes
and
to
get
to
these
successful
releases.
There's
a
lot
of
testing
that
happens.
A
If
you've
never
visited
the
kubernetes
test
grid,
it's
actually
pretty
inspiring
how
many
tests,
how
rigorous
those
tests
are
and
against
all
the
platforms
that
we
do.
It
I
think
there's
about
like
20
or
so
different
configurations
and
platforms
that
end
up
in
the
test
grid
and
that's
not
counting
all
the
other
projects
around
kubernetes
that
also
test
it
outside
of
just
a
normal.
Every
get
check-in
gets
this.
This
whole
service.
B
So
now
I'll
cover
our
shiny
new
process,
which
we
did
just
use
and
it's
a
you
know
getting
there.
It
totally
worked
in
the
end,
so
the
first
thing,
of
course,
is
the
bug
being
reported.
So
this
is
through
our
disclosure
process,
which
we
have
the
mailing
list.
We
have
all
our
individual
emails
if
necessary.
B
We
also
have
like
GPG
keys
there
as
well,
and
it's
on
you
know:
kubernetes
videos
watch
security
very
easy
to
buying,
hopefully
in
the
SEO
now
and
yes,
security
researchers
can
hopefully
get
right
there,
so
within
24
hours,
we'll
form
this
kind
of
fixed
team.
That
has
like
the
experts
in
this
area
that
the
bug
affects
so
like.
B
If
it's
in
you
know,
the
mig
mode
will
contact
whoever
we
need
to
contact
there
and
those
engineers
will
be
involved
from
the
start
like
discussing
fixes-
and
you
know
going
over
how
this
should
be
fixed,
how
severe
it
is,
and
everything
like
that-
and
this
will
all
be
done
on
a
private
repo
that
we
have
set
up
and
all
the
patches
are
posted.
There
they're
reviewed
there
and
everything
kind
of
happens
there
until
yeah.
B
So
at
this
point
well
also,
if
the
bug
is
critical
enough,
we'll
let
vendors
know
so
that
they
can
either
upgrade
early
or
even
like
review
patches
to
make
sure
that
it's
not
going
to
break
their
infrastructure.
That
way,
we
are
ensured
that
upgrades
are,
you,
know,
consistent
and
keeping
everything
so
working.
B
So
if
it's
super
severe,
we'll
definitely
do
this
and
get-and.
If,
honestly,
if
you
think
like
you,
your
vendor
should
be
on
this
bailius,
you
should
probably
find
one
of
us
after
and
so
that
we
can
make
sure
and
then
within
21
days,
will
cut
the
release
from
the
private
repos
and
then
post
the
patches
to
the
public
repo
right
after
the
binaries
are
available.
B
So
there's
actually
this
like
entire
spec
for
scoring
different
CVS
based
on
the
severity
and
right
now,
it's
like
version
3.0,
but
like
at
the
time
of
heartbleed.
It
was
2.0
and
it
takes
all
these
things
like
into
account.
So
like
attack,
vectors,
authentication,
confidentiality
so
like.
If
your
data
is
like
going
to
be,
you
know,
disclosed
and
that's
not
something
that
should
happen.
That's
definitely
there.
B
So
you
can
kind
of
score,
and
then
it
will
calculate
this
number
for
you
out
of
10,
so
just
to
give
some
examples
like
heartbleed,
and
this
was
on
an
old
scoring
system
which
is
I
almost
think.
It's
like
spurned
3.0,
but
I
was
like
reading
all
these
opinion
in
his
blog
post.
So
I
cannot
confirm
that
it
was
a
5
out
of
10,
which
is
weird
because,
like
heartbleed
was
super
bad
right,
so
I
was
like,
like
I
went
down
this
rabbit
hole
of
like
why.
B
So
it
turns
out,
like
the
old
scoring
method,
kind
of
didn't,
take
into
attack
that
and
attack
all
these
systems
like
so
many
people
had
to
upgrade
that
day.
It
was
like
a
huge
thing,
and
people
were
like
successfully
exploiting
it
on
people
systems
that
they
did
not
upgrade,
and
it
is
very
easy
to
exploit
so
I
think
they
after
this
for
like
we
need
to
like
redefined
the
wheel.
So
to
give
a
better
example,
dirty
cow
was
so
seven
point,
eight,
which
actually
kind
of
seems
like
it
fits
it
affected.
B
A
So
some
of
the
challenges
is
that
you're
responsible
for
non-public
work,
non-public
work
is
not
glamorous
and
open-source
communities.
Some
of
the
incentives
that
we've
come
up
with
is
that
one
you
can
say
that
you've
involved
in
the
team.
It's
publicly
documented
on
the
security
page,
that
this
product
security
team
is
responsible
for
the
successful
rollout
of
security
updates,
and
we
all
know
who
here
is
using
kubernetes.
A
We
all
know
how
important
it
is
intrinsically
to
the
entire
ecosystem.
The
other
bit
is
that
we
have
these
lead
positions
who
actually
are
in
charge
of
taking
the
vulnerability
from
disclosed
to
public,
and
they
do
get
credit
by
being
the
person,
email,
nails,
the
mailing
lists,
the
announce
list,
etc
about
the
about
the
lead
about
the
vulnerability.
A
The
big
thing
that
we
need
to
do
here
is
that
we
actually
don't
have
a
way
of
electing
a
new
product
security
team.
It
was
kind
of
just
made
up
of
the
ragtag
rogue
team
that
like
came
together,
and
we
all
just
said
we're
going
to
take
care
of
this
process,
but
it'd
be
really
nice.
If
we
had
new
ways
of
people,
volunteering
and
getting
involved
and
holding
them
accountable
to
ensuring
this
as
well
led.
B
A
A
We've
really
been
the
only
ones
to
audit
the
process
and
test
it
and
ensure
that
we're
using
as
many
leveraging
as
many
best
practices
as
we
can
find.
So
if
you
want
to
really
help
out,
the
first
is
that
we
need
you
to
all
join
kubernetes
announce
all
this
work,
as
Jeff
said,
is
meaningless
if
nobody
actually
upgrades
and
so
kubernetes
announces
where
you
actually
find
out
about
these
vulnerabilities,
the
next
bit
is
actually
reviewing
the
process.
A
A
So
just
actively
looking
and
seeing
if
the
way
that
kubernetes
is
being
managed
as
a
community
is
a
way
that
you
feel
like
it
should
be
managed
and
then
yeah.
Alright
I
already
gave
a
pitch
that
I'll
just
do
it
again
tested
for
repo
is
maintained
by
a
small
set
of
very
brave
engineers,
and
it
is
the
backbone
of
the
project.
I
think
that
we
would
not
be
here
today
if
kubernetes
was
seen
as
an
unstable
project,
a
lot
of
us
very
confidently.