►
From YouTube: Kubernetes Community Meeting 20180315
Description
We have PUBLIC and RECORDED weekly video meetings every Thursday at 10am US Pacific Time.
Notes: https://docs.google.com/document/d/1VQDIAB0OqiSjIHI8AWMvSdceWhnz56jNpZrLs6o7NJY
A
A
C
C
So
let's
say
you
set
up
a
new
kubernetes
cluster
and
you
run
it
out
in
production,
and
you
want
to
monitor
it
and
make
sure
everything's
healthy.
You
probably
are
gonna
set
up
something
like
metrics
collection,
so
you
know
that
your
cluster
is
not
over
underutilized
health
checks,
so
you
know,
endpoints
are
actually
responding
correctly
and
coming
as
everything
is
a-okay.
C
They
could
happen,
but
I'm
here
to
unfortunately
tell
you
that
your
cluster
can
still
be
broken,
and
we
found
this
out
the
hard
way
by
running
clusters
in
production
at
Comcast
and
we're
running
those
globally
around
the
world,
not
just
in
America,
and
basically
your
cluster
can
still
be
broken
in
the
fact
that
if
a
developer
goes
to
create
and
deploy
something,
your
pod
gets
stuck
in,
the
creating
state
could
get
stuck
in
the
terminating
state
could
be
you
know,
unable
to
mount
disks.
It
could
be
unable
to
release
its
IP
allocations
with
the
C&I.
C
A
lot
of
stuff
could
be
completely
broken.
So
one
thing
we
found
out
that
would
turn
up
some
of
these
issues,
even
though
we
had
all
these
other
types
of
monitoring
was
running,
cops
validate,
which
cops
is
an
awesome
project,
and
we
use
that
here
at
Comcast
to
do
all
the
baseline
cluster
bootstrapping,
but
cops
validate
was
really
good
at
showing
these
errors,
because
they
checked
a
lot
of
really
meaningful
things
inside
of
the
kubernetes
cluster.
So
what
we
did
is
it
kind
of
took
that
idea
of
cops,
validate
and
turned
it
into
a
monitoring.
C
Endpoint
excuse
me
of
kind
of
cold
by
the
way
of
a
cough
drop.
We
turn
it
into
a
monitoring
endpoint
that
very
simply
tells
you
your
cluster
is
up
or
there's
something
wrong
with
it,
and
if
there's
something
wrong,
we
give
you
basically
some
some
explanation
about
what
we've
detected
wrong.
So
uber
healthy
verifies
that
a
pod
can
be
deployed
to
every
node
in
the
cluster.
We
do
that
through
a
daemon
set
as
optionally,
a
small
EBS
or
persistent
volume
attached
to
it,
then
we
verify
that
that
daemon
set
can
be
torn
down
appropriately.
C
All
the
pods
can
be
terminated
can
all
be
cleaned
up
appropriately.
Everything
comes
back
to
normal.
We
also
look
at
the
cube
system
namespace
to
make
sure
that
your
pods
aren't
restarting
what
we
would
call
too
often,
which
for
us
right
now
is
I
think
three
times
in
ten
minutes
or
something
like
that,
but
that
well
obviously
be
tunable.
We
also
want
to
make
sure
that
pods
are
all
in
the
ready
phase
when
you
would
expect
them
to
be
in
your
cluster.
So
when
I
know
it
has
been
online
for
more
than
10
minutes.
C
I
would
expect
that
all
of
the
daemon
sets
and
things
running
on
that
node.
Well,
everything
in
the
cube
system,
namespace
on
that
node
should
be
in
the
ready
phase
of
the
pod
lifecycle,
and
then
we
also
check
component
statuses
to
make
sure
those
are
always
online,
because
those
are
obviously
pretty
important
when
you
have
that
CD
and
the
scheduler,
and
things
like
that.
So
here's
a
diagram
that
looks
more
complicated
than
it
actually
is.
C
C
C
C
So
that's
what
the
struct
under
it
looks
like
our
lambda
checks
that
JSON
blob,
if
that
JSON
blob
goes
down,
starts
showing
an
error
message,
then
our
lambda
will
figure
that
out
and
we
want
to
open
source
this
lambda
with
the
project
as
well,
not
necessarily
with
our
deployment
tools
that
deploy
the
lambda
but
the
lambda
itself,
so
that
you
have
the
option
of
using
it
and
then,
of
course,
it
can
send
alerts
to
any
integration
that
you
need,
such
as
in
this
situation.
This
was
Pedro
duty.
C
This
sends
to
slack
so
we're
using
it
in
production
at
Comcast.
You
have
this
in
four
different
regions
around
the
world
and
at
least
six
clusters
I.
Consider
it
alpha
right
now
before
we
actually
recommend
this
for
general
availability.
We
want
to
make
sure
100%
that
all
checks
are
actionable
from
which
is
one
of
the
main
goals
of
the
project,
but
we
want
to
keep
working
on
that.
C
We
want
to
add
some
more
checks
like
service
testing
and
checking
Kubb
dns,
and
then
we
need
to
document
it
a
lot
better
for
public
consumption.
So,
finally,
if
you
have
any
questions,
it's
super
quick
presentation,
probably
best,
because
I
may
kick
the
bucket
soon.
You
can
email
me
agree
at
Comcast
or
on
slack
I'm,
just
airing
Grier
on
that
you
actually
can't
have
spaces
in
your
slack
username,
so
terminates
like
Eric
Grier
or
throw
me
an
email
and
I'm
also
going
to
plan
on
reaching
out
to
request
this
for
sandbox
consideration.
A
A
And
if
I
actually
read
the
first
bullet
teeth,
point
of
this
is
notes.
It
says
the
release
team
is
in
a
meeting
right
now,
so
the
full
status
is
in
the
kubernetes
community
meeting
notes,
but
due
to
some
security
releases
and
scalability
testing
issues,
we
push
the
release
from
March
21st
back
to
March
26
and
plan
to
lift
code
freeze
by
end
of
day
money.
Monday,
assuming
everything
goes
to
plan.
It
looks
like
they're,
also
looking
for
a
release.
1.11
lead
so
take
a
look
at
the
role
description.
A
B
So
I
just
wrote
down
some
notes
beforehand
to
go
over
there
in
the
document
for
anybody
who
wants
to
follow
them
and
I
also
just
posted
them
into
chat
for
if
you
want
to
follow
my
notes
as
I
go
through
them,
so
sig
off
has
been
pretty
busy
over
the
110
release
and
I'm
gonna
go
through
some
highlights
of
some
things.
That
I
think
are
a
significant
over
the
110
release
and
then
talk
a
little
bit
about
some
things
that
we
want
to
be
doing
in
the
future.
So
first
up
is
pod
security
policies.
B
For
anyone
who
doesn't
know
what
pod
security
policies
is
in
kubernetes
a
pod,
an
individual
pod
has
a
lot
of
ways
of
actually
getting
around
some
of
the
container
technology.
Is
that
you'd
expect
when
you've
run
out
of
regular
docker
container,
so
you
can
run
a
privilege
pod,
you
can
run
a
pond
with
hostnet
or
key.
You
can
run
a
pod
that
mounts
arbitrary
volume
this
into
your
container
and
well.
This
is
very
helpful
for
things
like
node
agents.
It's
a
very,
very
obvious
hole
when
you
think
about
multi
tens
that
use
cases
scenarios.
B
So
this
is
now
in
a
policy
v1
beta
1
we're
looking
for
a
user
feedback
as
we
try
to
push
this
towards
GA,
and
we
would
like
to
sort
of
roll
this
out
in
the
same
way
that
we
have
previously
rolled
out
things
like
barback,
pushing
it
to
the
point
where
a
users
sort
of
come
to
expect
this
as
a
default
way
that
enables
security
in
the
cluster.
The
second
feature:
I'll
talk
about
is
advanced
auditing,
so
the
API
server
actually
has
many
auditing
features.
B
Inside
of
it
that
allows
you
to
inspect,
who
did
what
look
at
what
type
of
requests
came
in?
Who
was
issuing
them?
What
the
response
was
and
even
look
at
the
exact
body
of
the
request
and
its
result?
What
we
are
we,
this
is
still
going
to
be
beta
in
110,
but
we
are
looking
to
push
it
to
stable
soon,
a
majority
of
the
issues
that
were
fixed
over
110
involved
with
scalability
issues.
B
Google
has
been
doing
a
great
job
in
terms
of
testing
it
out,
putting
it
through
its
paces
and
finding
interesting
bugs
and
just
running
this
at
scale.
So
that
has
been
the
primary
focus
of
the
beta
efforts
and
over
the
next
couple
releases
we
will
be
looking
to
push
this
to
stable,
big
shot
to
mick,
who
has
been
doing
a
great
job,
just
keeping
a
record
of
the
features
that
are
going
in
and
every
release.
B
If
you
look
at
issue
number
five,
eight,
zero,
eight
three,
that's
a
good
example
of
sort
of
him
maintaining
what
the
future
status
is
and
if
you
ever
have
a
question
about
what
the
features
look
like.
You
can
find
those
issues
and
they're
very
detailed
talking
a
little
bit
about
some
of
the
Alpha
features
that
were
introduced.
B
This
release,
the
Maya,
probably
probably
the
most
interesting
feature
that
we've
been
working
on
in
sort
of
association
with
the
container
identity
working
group,
is
something
called
the
token
request,
API
in
kubernetes
when
you
deploy
a
container
and
deploy
a
a
deployment
that
spawns
out
a
bunch
of
containers.
Those
containers
get
a
service
account
that
can
talk
to
the
kubernetes
api.
B
The
problem
with
service
accounts
is
that
one,
their
stores
and
secrets,
so
anything
that
can
read
secrets
can
now
grab
those
credentials
and
escalate
to
the
most
privileged
service
occurring
in
a
particular
namespace
and
in
addition,
when
you
deploy
something
like
a
deployment
and
you
get
three
containers
out
of
that
or
pods
out
of
that,
all
of
those
pods
have
the
exact
same
service
account.
They
are
unable.
You
are
able
to
differentiate
between
any
particular
one.
B
The
token
request
API
is
a
way
of
dynamically
creating
service
accounts
that
does
not
store
them
in
secret,
so
you
will
send
a
request
to
the
API
server
and
say:
I
would
like
a
surface
account
for
this
particular
service
or
service
account,
and
the
API
will
actually
dynamically
sign
that
give
it
back
to
you
and
that's
no
longer
stored
in
secret.
They
have
exclusive
experience.
They
will
be
findable
to
specific
pods,
so
they
will
actually
contain
information.
B
That
says
this
service
account
is
not
just
only
for
this
name
service
account
or
this
credential,
but
this
is
actually
for
that
specific
pod
running
on
that
note,
if
that
pod
is
ever
deleted,
that
credential
will
just
completely
expire,
it
immediately
and
no
longer
be
valid
and,
in
addition,
you'll
be
able
to
create
audiences
tokens
with
audiences
for
things
other
than
the
API
server.
That's
sort
of
a
technical
way
of
saying
you'll
be
able
to
create
things
that
will
not
be
valid
for
the
API
server,
but
it
will
be
valid
for
an
extra
servus.
B
So
a
great
example
of
this
would
be
something
like
hash,
eforp's,
vault,
kubernetes
plugin.
When
you
use
hash
you
crimson
vault,
what
it
allows
you
to
do
is
take
a
service
account
token
and
exchange
it
with
vault
for
a
secret,
and
this
is
a
great
way
of
basically
plugging
into
an
external
secret
store.
The
problem
with
that
is
that,
when
you
hand
over
that
token,
that
token
is
also
valid
for
the
API
servers.
So
that's
a
bad
pattern
in
our
perspective,
and
we
want
to
create
tokens
that
are
only
valid
for
specific
use.
B
Cases
like
this
is
only
valid
for
false,
so
the
combination
of
this
feature-
it's
an
alpha
right
now
over
the
next
few
releases,
we'll
be
pushing
it
to
beta
getting
it.
So
the
coolant
has
the
ability
to
automatically
request
these
and
insert
them
into
your
pods
and
I
think
that
the
future
of
this
is
sort
of
looking
like
something
where
kubernetes
fits
much
better
in
with
integrating
with
external
secret
stores,
and
we
can
sort
of
start
to
move
away
from
the
existing
secrets
which
have
issues
around
ACLs
and.
B
The
next
one
is
client
go
external
credential
providers.
We
have
a
lot
of
ways
of
integrating
with
external
IDPs,
but
we
do
not
have
a
good
client
side
story
for
that
today.
At
all
client
side,
custom
code
is
compiled
into
plain
go
and
as
of
1-10,
we
will
introduce
for
introducing
the
ability
to
allow
people
to
write
client-side
plug-ins,
to
do
things
like
log,
in
a
credential
rotation
for
coop,
CTL
and
so
on
and
so
forth.
B
A
really
interesting
use
case
is
for
TLS
boot
stuffing,
because
this
will
be
a
client
know
feature
and
not
just
a
QTL
feature.
The
couplet
will
also
allow
this,
so
you
could
consider
a
world
where
the
couplet
could
create
a
initial
CSR
as
its
own
AWS
instance
like
that
abhi,
for
example,
by
writing
a
custom
plug-in
that
uses
the
google
it,
and
this
helps
us
get
past
some
of
the
bootstrapping
issues
that
we'd
see
internally
by
not
being
able
to
have
those
kind
of
integrations.
B
Protocols
that
we
don't
integrate
with
like
LDAP
and
so
on
and
so
forth,
encryption
at
rest
also
got
an
external
kms
integration
or
external
integration.
Encryption
at
rest
is
the
mechanism
for
the
kubernetes
api
server
to
encrypt
resources
before
it
stores
it
in
SED.
A
really
obvious
use
case
of
this
would
be
encrypting
secrets,
so
I
want
to
encrypt
secrets
before
I
start
of
an
SUV.
This
allows
you
to
do
that.
B
So
I
will
talk
about
it,
a
little
bit
with
a
grain
of
salt,
but
there
has
been
some
initial
exploration
of
a
bounty
for
kubernetes.
This
is
not
specifically
owned
by
sig
off.
In
fact,
it
will
need
owners
from
far
more
states
that
just
us,
but
was
discussed
in
sig
off
and
I
think
is
interesting
for
people
that
are
interested
in
that
kind
of
thing.
B
It
is
very
easy
for
us
to
sort
of
say
if
you
find
a
vulnerability,
yet
only
if
you
turn
all
these
security
features
on
the
it's
a
valid
one,
but
we
also
want
to
make
this
real
estate
into
actual
clusters
that
are
running
so
follow
up
on
that
and
the
forum's
other
people.
That
may
have
more
information
for
that
particular
item,
but
that
it
for
my
signal,
update
and
if
anyone
has
any
questions,
I'm
happy
to
field
them.
A
E
Hello,
can
you
hear
me
yep
monochrome,
so,
what's
going
on
in
sickest,
imitation
first
important
update
is
that
we
have
a
new
cichlid
Fraternity
branching
from
Korres
agreed
to
become
music
lick.
It
he
replaced
Fabian
Hertz.
All
certain
colors
I
would
like
to
thank
Fabian
for
leading
the
seek
for
a
year
and
half
so
far
for
all
great
contributions.
E
It's
implemented
in
an
adapter
pattern
so
that
we
are
providing
only
API
and
if
you
want
to
you,
know,
have
a
real,
improve
a
implementation
integrated
with
your
system,
for
example,
which
from
me
use
with
cystic
dated
or
any
other
monitoring
systems.
You
need
to
implement
this
adapter
I
am
aware
of
the
work
to
provide
integration
with
stackdriver,
not
sure
how
about
chromatids
and
other
systems.
E
E
E
E
First
is
to
focus
on
stabilize
stabilizing
things,
so
we
want
to
graduate
the
API,
so
we
have
to
G
a
master
matrix,
API
custom,
matrix,
API
external
matrix
API.
We
want
to
also
stabilize
matrix
server
a
bit
and
duplicate
hipster.
Apart
from
that,
there
is
a
plan
to
define
historical
matrix
API,
so
that
component
interested
in
using
historical
data
to
perform
some
actions
like,
let's
say,
scheduled,
air
or
vertical
cut
out
the
Skyler
can
can
consume
them
from
multiple
systems
there
is.
E
There
is
an
idea
to
have
to
have
agreement
on
logging
architecture
and
vision.
Three
years
ago
or
getting
half
ago,
there
was
monitoring
architecture,
vision
proposed
and
agreed
within
the
community.
This
is
this.
This
was
a
really
great
achievement
of
cig
instrumentation,
because
it
was
a
foundation
for
many
design
decisions
and-
and
basically
you
know
some
decisions
made
in
the
past
and
will
be
a
foundation
for
such
decisions
in
the
future.
E
We
started
discussions
about
exposing
cubelet
health
health
status.
This
would
be
very
useful,
for
you
know,
monitoring
the
state
of
cubelet,
but
this
is
different
from
matrix
endpoint.
So
we
need
to
figure
it
out
how
to
do
this,
and
we
have
also
on
a
plate,
some,
let's
say
organizational
work.
We
would
like
to
move
all
sig
instrumentations
projects
to
a
new
home.
A
Alright,
it
doesn't
look
like
we
have
any
questions
and
yeah.
Thank
you
very
much.
Thank
you.
So
next
up
we
have
some
announcements,
so
registration
for
the
kubernetes
contributor
summit,
which
takes
place
just
before
coupon,
is
now
open.
So
if
you
are
interested
in
attending
that,
please
check
the
link
in
the
evening
notes.
It
is
separate
registration
from
coupon
itself,
so
you
have
to
register
for
both
if
you're
interested
next
week,
we
have
office
hours,
we're
looking
for
volunteer
developers
to
help
answer
questions
that
people
are
asking,
there's
a
link
for
that
as
well.
A
For
more
information
in
the
meeting
Oates
we
have
the
the
videos
from
the
helm
summit
have
been
posted,
there's
also
links
to
the
hose
they're,
all
on
YouTube,
there's
a
link
to
the
YouTube
playlist
and
finally,
for
this
week
we
have
some
shout
outs.
So
shout
outs
are
the
section
of
the
kubernetes
community
meeting
where
we
acknowledge
awesome
work
done
during
the
past
week
by
various
members
of
the
community.
A
A
D
Are
you
referring
to
the
note
the
note
I
put
in
the
comment
stream?
Yes,
I
am
yes,
so
I
think
looking
for.
We
may
be
able
to
resource
this
out
of
the
six
scale
group,
but
thought
it
would
be
worth
mentioning
that
one
of
the
things
that
you
mentioned
it
briefly
briefly
previously,
but
we
really
are
looking
for
to
add
someone
officially
to
the
release
team
to
try
to
track
large
scale
regressions
at
the
moment,
this.
D
Those
tests
take
so
long
to
run
that
if
we,
if
we
make
them
blocking,
it'll
reduce
velocity,
so
we
really
need
a
human
being
to
be
on
top
of
making
sure
that
when
those
regressions
happen,
they're
driving
back
the
work
that
needs
to
happen
earlier
in
the
release
cycle.
So
we
don't
have
a
slip
at
the
last
minute,
so
I
would
just
call
it
a
call
for
volunteers
contact
me
on
slack
or
offline.
Somehow,
if
you're
interested
in
that.
A
D
Maybe
maybe
I
should
add
one
more
comment
there,
which
is.
This
is
probably
a
good
role
for
someone
who
is
looking
for
a
way
to
make
a
newer
contribution
on
the
project,
because
it's
really
a
little
bit
more
of
a
coordination
role.
So
I'd
say
you
don't
have
to
be
a
scalability
expert
to
make
a
big
contribution
here.
Thanks.