►
From YouTube: Kubernetes kops office hours 20201218
Description
Recording of the kops office hours meeting held on 20201218
A
Hello,
everybody
and
welcome
to
the
chaos
office
hours.
I
am
your
moderator,
facilitator,
justin,
santa
barbara.
I
work
at
google
today
is
friday
december
18th.
This
is
this
will
be
our
last
meeting
of
2020..
We
do
have
a
couple
of
things
on
the
agenda.
A
Please
do
feel
free
to
add
things
to
the
agenda
if
you
would
like
to
and
add
your
name,
if
you
would
like
to,
we
have
a
relatively
light
festive
attendance
today,
but
I
think
we
can
just
dive
right
in
the
first
thing
on
the
agenda
is
that
I
was
supposed
to
review
the
std
terraform
resource
name
issue.
I
did
not
do
that,
but
I
shall
do
that
immediately
following
this
meeting.
A
I
don't
know
if
there
were
any
other
action
items
last
time
that
that
we
were
supposed
to
review.
B
C
A
Pleasure
so
the
first
item
on
the
agenda
on
the
main
discussion
agenda
is
from
olay,
who
I
don't
see
here
today,
but
a
pr10439
to
review
the
node
authorizer
in
1.20..
I
don't
know
if
anyone
can
speak
to
this.
B
I
discussed
with
him
a
bit
about
this
from
what
I
know.
No
daughterizer
role
was
moved
to
cops
controller,
or
is
it
still
used
in
any
form
at
the
moment,.
A
I
think
it's
only
used
on
aws.
That
is
the
other
thing
on
my
agenda
for
this
afternoon,
is
to
look
at
getting
support
for
that
on
gcp
going.
A
I
will
certainly
take
a
look
at
this
as
well
and
see
what
the
status
is,
because
if
I
recall
correctly
john,
we
are
always
using
node
authorize
on
aws.
It's
not
even
feature
flagged.
D
But
then
we
yeah,
no,
I
think
there
was
a
there's,
a
no
authorization
mode
which
I
think
works
anywhere,
but
so
you
can
have
no
authorization,
authorization.
B
A
Okay,
well,
yeah,
I
think
definitely
it's
the
it's
the
direction
we're
going
in.
I
don't
know
about
timing.
I
feel
like
getting
more
support
into
the
awesome.
Getting
support
for
more
clouds
into
the
new
node
authorizer
pipeline
would
be
good
and
would
be
a
good
directional
thing.
D
B
A
A
Yeah
we
should
we
can.
I
can
also
check
with
the
the
people
that
originally
added
that
to
see
if
they're
still
using
it.
A
Okay,
next
item
is
from
merrell
three
about
etcd
manager,
an
sd
manager
bug.
I
why
I
presume
yes,
I
will
take
a
look
at
what
this
is.
B
I
think
he
did
a
lot
of
digging
in
city
manager.
He
added
some
more
comments
and
his
research
on
the
issue
of
upgrading,
so
I
think
he
explained
how
to
reproduce
it.
A
The
certificate
change
in
go
115
where
they
decided
to
enforce
like
mark
old
certificate
like
start
requiring
the
cert
to
be
in
the
alt
subject:
alternate
alternative
names
instead
of
the
cn
but
yeah
I'll.
Take
a
look
at
this
as
well.
B
A
And
I
think
I
I
think
I
I
think
I
put
in
a
fix
to
get
that
to
pass,
but
we'll
see
whether
it's
actually
passing
and
but
yeah
the
the
migration
is
going
to
be
interesting.
A
B
Okay,
demigration,
we
all
know
it
will
happen
somehow,
and
I
was
thinking
besides
the
test
grid
changes
it's
a
bit
overwhelming
at
the
moment
with
all
the
tabs
and
they
run
pretty
rarely
to
to
make
sense
of
the
results.
So
a
bit
on
the
side.
As
a
side
note,
I
don't
think
quickly
tests
help
that
much
there
are.
I
mean
it
would
help
if
they
would
always
work
without
flakes,
but
unfortunately
there
is
a.
B
I
don't
know
25
chance
for
something
to
fail,
or
I
don't
know,
maybe
a
little
smaller
and
then
no
one
will,
and
you
also
see
only
the
two
weeks
of
results,
because
that's
the
configuration
a
bit
hard
to
change
that.
I
think
peter
tried
and
couldn't
do
much.
So
my
proposal
would
be
to
change
our
periodic
tests
from
docker
to
container
d.
B
B
I
don't
think
that
would
add
too
much
too
many
runs
because
we're
not,
as
let's
say
popular
as
kubernetes,
and
I
don't
know
if
everything
works
default
to
for
new
clusters
in
120.
So
only
new
clusters
and
121
move
everyone
to
container
d.
A
A
Test
so
on
the
grid
test,
the
I
don't
know
if
I
can't
remember
who
merged
it.
Thank
you
to
have
emerged.
I've
started
annotating
the
jobs
with
the
parameters
and
the
intent
is
that
I
want
to
start
to
capture
the
results
like
the
the
configuration
and
the
results
separately
from
test
grid,
because
I
agree
like
there
was
the
test
grid
results.
The
the
results,
as
shown
in
test
grid,
are
like
they're
nice
to
drill
down.
A
If
you
want
to
like
click
on
one
particular
test,
but
they
aren't
really
reflective
of
the
the
grid
approach,
which
is
you
know
that
the
the
tests
are
not
individually
interesting.
They
are
only
interesting
in
aggregate,
so
you
have
to
like
slice
across
all
the
configurations
and
say
like
which
is
the
most
likely,
which
is
the
top
configuration
that
that
flakes
and
then
look
at
that
across
multiple
ones.
A
So,
in
other
words
like
I
did,
I
did
an
early
like
analysis
of
this
and
I
think
we
can
say,
like
you
know
like
for
the
sake
of
argument
using,
I
think,
actually
like
rel,
seven
and
or
rel.
Six
and
psyllium
doesn't
work
or
something
like
that,
and
that
was
like
one
of
the
top
something
some
combination
like
that
was
one
of
the
top
like
factors
and
so
like
deciding
what
we
do
about
that
like
either
excluding
it
because
it
just
can't
be
fixed
or
fixing
it.
A
So
that's
why
the
that's!
What
that's
sort
of
where
I'm
going
with
the
grid
is
to
to
try
to
gather
information
in
aggregate
that
we
don't
see
on
the
individual
runs.
We
can
also
retain
those
results
for
longer
by
mirroring.
F
Yeah
yeah,
if,
if
we
could,
if
we
could
capture
that
those
results,
then
we
could
actually
query
them.
You
know
stick
them
into
some
database
and
actually
provide
some
useful
details
about
them
and
then
pull
the
data
out
when
we
want
to
actually
run
a
test
that
is
flaky
exactly
so.
I
think
I
think
that
makes
a
lot
of
sense.
D
D
I
think
we
might
just
want
to
pick
a
version
and
say
when
kubernetes
is
greater
than
this
version
you
default
to
community.
A
Yes,
the
the
this
is
sort
of
what
the
upgrade
thing
is
supposed
to
deal
with
right
is
to
sort
of
help
you
originally
like
back
in
the
day.
This
is
what
the
cops
have
breakdown
I'm
supposed
to
do.
It's
like
get
you
off
a
locked
configuration
that
is
likely
wrong.
A
B
A
A
B
That
to
the
only
bigger
issue,
is
the
docker
configuration.
So
maybe
we
don't
change
people
that
have
docker
settings.
B
A
A
And
yeah,
I'm
fine
with
moving
the
the
periodic
jobs,
the
the
more
rapid
periodic
jobs,
the
the
named
periodic
jobs,
I
guess
to
to
container
d.
A
I
can't
remember
whether
we
put
in
we
we
didn't
yet
did
we
double?
Did
we
do
cri?
Did
we
do
container
d
and
docker
in
the
grid
or
not?
I
can't
remember
whether
I
actually
did
better
yes,
cool
yeah.
B
B
A
A
We
also
can
like
separate
from
the
grid
is
the
idea
of
generating
those
jobs.
I
thought
there
was
another
script
to
generate
some
other
ones,
so
we
could
look
at
like
whether
we
want
to
do
that
whether
it
would
make
the
maintenance
of
those
jobs
easier.
I
don't
know
whether
it
would
or
would
not.
A
The
thing
well,
I
can
have
a
look
at
doing
that
then
120
jobs
from
script,
because
I
can't
remember
whether
there
is
a
script
or
not.
Alongside
go
great,
I
feel
like
there
is,
but
I
don't
I
can't
recall,
but
this
would
be
testing
the
latest
kubernetes
with
sorry.
The
latest
k-ops
with
kubernetes
120.
A
The
grid,
by
the
way,
does
in
amongst
its
ridiculous
traversal
of
impossible
combinations,
test,
skewed
kubernetes
and
chaos
versions.
I
don't
recall
whether
it
like
goes
up.
I
don't
think
it
does.
I
think
there
is.
B
A
catch
there
that
we
try
to
test
cops
versions
with
higher
kubernetes
versions
that
I.
A
A
Okay,
I
can
look
at
that
like
grid.
There
is
the
the
environment
variable
grid
with.
D
B
A
So
then,
on
the
container
d,
migration,
it
sounds
like
there's
a
decision
to
be
made.
I
guess
at
our
next
meeting
or
not
so
timeframe
about
whether
we
would
like
to
change
the
default
if
we're
going
to
change
the
default
across
the
board,
whether
we
want
to
change
the
default
m120
or
121.
A
B
A
Yeah,
I
can
have
a
look
at
that
as
well.
I
think
there
it
looks
like
there's
one
test,
which
is
always
failing
yeah
true.
A
But
yeah
it's
it's
yes.
I
agree,
there's
no
real
pattern
here.
It
may
be
timing.
One
thing
we
can
try
is
to
beef
up
the
to
use
bigger.
We.
B
A
B
B
A
B
H
Conformance
looks
like
it's
using
the
ci
latest
release
marker
for
kubernetes,
so
it's
using
121
alpha
zero.
Maybe
we
could
move
that
down
to
a
more
stable
version.
B
It
happened
all
the
time.
If
you
look,
we
have
one
of
the
other
two
jobs,
that's
running,
also
on
ci
latest
and
one
on
ci
stable.
So.
A
There's
a
really
weird
there's
a
really
weird
label:
selector,
okay,
the
articulator-
it
looks
like
one
of
the
names
instance
names
is
too
long,
and
but
anyway,
I
will
take
a
look.
Oh
yeah.
That's
that
true!
A
Okay,
sorry!
Next
on
the
agenda,
I
was
me
so
there
are
hello
there
are.
I
sent
I
updated
my
add-ons
pr,
that's
been
up
for
a
long
time
and
I
also
sent
another
one,
the
two
of
them.
The
first
one
is
basically
based
on
primarily
on
john's
excellent
feedback
around
the
security
model,
and
so
the
the
change
we've
made
in
or
we
can
make
in
cluster
add-ons
is
to
effectively
pre-create
the
rbac.
A
So
the
operator
itself
doesn't
need
to
create
the
our
back
roles,
and
so
the
the
operator
needs
vastly
fewer
permissions
still
a
couple
more
than
I
would
like,
but
vastly
fewer.
So
I
think
I
think
you
will
be
more
happy
john.
In
that
regard.
The
next
one
is
might
make
you
even
more
happy,
which
is
retain
sort
of
the
execution
model.
Today,
where
we
run
the
operator
in
a
sort
of
one-shot
model,
we
we
run
it
client-side
and
I'll.
A
Come
back
to
that
in
a
second,
we
run
it
in
in
the
cop
cli
and
we
basically
find
out
the
manifest
that
will
be
created,
and
so
it
goes
into
the
the
bootstrap
yaml
as
if
there
was
no
operator
involved.
There's
no
operator
that
runs
at
all.
There
are
probably
other
modes
in
between,
but
those
are
the
sort
of
ones
I
think
are
most
interesting.
The
the
huge
downside
of
the
pre-running
is.
A
It
now
requires
us
to
run
docker
on
as
part
of
chaops,
so
that
has
two
gotchas,
one
of
which
is
docker
and
docker
is
still
not
a
thing
or
still
a
thing
and
the
other
one
of
which
is,
it
would
be
nice
like
it.
It
would
be
nice
to
not
have
a
dependency
on
docker
at
all.
There
may
be
things
we
can
do
there,
but
that's
that's
the
that's
the
gotcha
there.
A
So
it's
it's
a
good
model
in
that
regard.
It
gets
rid
of
our.
It
gets
a
good
model
in
that
it
gets
rid
of
the
it
enables
us
to
update
add-ons
without
updating
without
having
to
update
the
chaos
binary
and
it
enables
them
to
have
processing
complicated
processing.
So
that's
good.
I
think
the
downside
is
the
need
for
docker
execution
in
that
mode.
It's
not
required
everywhere
else.
A
It's
only
required
in
that
mode,
but
that
is
a
downside,
and
I've
been
like
pondering
how
we
can
make
it
better,
like
you
know,
like
we
can
run
in
g,
visor
or
rootless
containers
seem
to
be
coming
and
may
actually
not
require
docker
installed
and
may
run
in
a
container
that
sort
of
thing,
so
that's
an
option
or
like
just
run
it
in
a
root
which
isn't
great
so
that
sort
of
those
sort
of
things
are
what
I'm
thinking
about.
A
But
anyway,
if
people
want
to
look
at
it
and
comment
on
it
that
would
be
greatly
received.
I
don't
think
either.
One
of
them
is
like
technically
ready
to
merge
like.
I
think
they
both
have
conflicts,
but
the
idea
should
be
there
and
you
can
sort
of
see
the
the
the
our
back.
Permissions
were
able
to
whittle
down
to
by
pre-creating
the
rbac
and
you're
able
to
see
the
the
docker
execution.
A
So
any
comments
greatly
received
any
comments
on
that.
Otherwise,
that's
the
last
item
in
our
open
discussion.
So
if
anyone
wants
to
add
an
bring
up
a
topic,
please
do
so
otherwise
we'll
go
on
to
recurring
topics.
A
Excuse
me
all
right.
First
item
on
the
recurring
topics
is:
when
should
we
have
the
next
informal
meeting
and
plans
for
the
january
1st
meeting,
I
think
we
pre-talked
about
the
plans
for
the
january
1st
meeting,
which
is
that
we're
not
going
to
have
one,
and
so
our
next
meeting
would
therefore
be
on
the
15th
ish
of
january,
and
I
think
we
should
probably
therefore
have
an
informal
meeting.
A
I
personally
am
going
to
take
a
little
time
off
in
between
that,
so
I
prefer
it
to
be
after
the
first
yeah
so
either
before
christmas
or
after
the
first
it
could
be
good
to
do.
We
could
try
to
do
an
inform
one.
I
don't
know
how
would
people
rather
do
something
next
week
before
christmas,
which
probably
means
monday
or
tuesday?
A
Well,
I'm
struggling
to
do
the
math?
Yes
monday
or
tuesday,
I
guess,
or
would
people
rather
do
something
after
the
first
or
both.
A
A
A
B
Okay
and
you
get
it,
you
got
it
right.
A
Fifth
january
5th
so
january
5th,
at
9am
we
will
do
an
informal
one
and
otherwise
we'll
have
our
next
formal
one
at
on
january
15th
at
noon,
eastern
9
a.m,
pacific.
A
Okay
and
then
we
have
nothing
planned
on
nothing
yet
planned
for
releases
for
the
upcoming
weeks,
but
we
have
a
discussion
of
18
119
and
120..
We
have
a
118
3
with
a
couple
of
blocking
prs,
which
I
think
the
second
one
is
the
scd
terraform
naming,
which
I
will
look
at
and
the
first
one
is
a
sabrina
is
around
backboarding.
The
target
group
related
fixes.
B
Yes,
those
things
were
breaking
attaching
target
groups
to
clusters,
which
I
don't
know
for
me.
It's
an
important
feature.
I
guess
other
people
use
it
too
and
I
think
it's
fixed.
I
would
appreciate
the
review
from
peter
and
rodrigo
and
without
with
this,
and
the
cherry
picks
of
whatever
we
decide
for
hcd
118
can
be
released.
A
B
H
So
the
the
gist
is
that
terraform
0.12
no
longer
allows
resource
names
to
begin
with.
A
digit
under
certain
scenarios
cops
create
cluster,
would
populate
xcd
member
names
to
start
with
a
digit.
If
the.
If
the
digit
is
the
first,
is
it
like
the
unique
part
of
whatever
the
full
cluster
name
is
or
something
like
that
for
the
zone.
H
And
so,
if
the
icd
members
start
with
a
digit,
then
their
terraform
ebs
volumes,
resource
names
start
with
a
digit
and
paraform
0.12
doesn't
like
that,
and
so
we
talked
about.
Can
we
change
the
member
names
on
an
existing
cluster
without
disruption,
and
I
think
that
impacts
how
etcb
manager
does
its
discovery?
H
And
so
I
don't
think
there's
a
documented
way
to
do
that.
Maybe
justin
has
ideas
or
we
changed
just
how
the
terraform
resource
name
is
generated
in
those
scenarios
where
the
name
would
otherwise
start
with
a
digit,
and
then
users
that
are
impacted
by
this
would
need
to
terraform
state
move
the
volume
from
the
old
resource
name
to
the
new
resource
name.
The
risk
of
that
is
if
they
upgrade
cops
to
the
version
that
includes
this
fix,
generate
the
terraform
output
and
then
terraform
apply
without
reading
the
upgrade
instructions.
H
Terraform
will
destroy
their
ebs
volumes
and
you
know
render
the
cluster
useless.
We
don't
have.
Terraform
does
have
prevention,
better
measures
to
prevent
accidental
deletion
of
ebs
volumes
and
stuff,
but
we
don't
have
that
set
already,
and
I
believe
that
it
needs
to
be
set
before
the
apply
for
it
to
take
effect.
A
I
don't
I
I
don't
know
how
we
can
make
that
more
acceptable
for
more
palatable
for
users.
We
could
what,
if
we
refuse
to
upgrade
to
terraform
12
if
they
have
an
if
they
have
this
problem.
A
A
A
B
You
know,
I
think
I
saw
somewhere
in
logs
that
it
actually
tries
to
to
do
a
to
do
a
restore
automatically
if
every,
if
it
finds
backups
and
all
volumes
are
empty.
A
A
Okay,
let's
talk
about
the
1
19,
then
it
looks
like
all
of
them
are
mostly
on
me
or
against
me.
The
blockers
I
just
added
in
the
370
371.
A
Okay,
cool
to
at
least
look
at,
and
I
will
try
to
do
this
this
afternoon
as
well.
D
A
Yeah,
we
don't
have
many.
I
can
try
to
share
this
screen
just
so.
We
can
quickly
look
at
them
see.
How
do
I
do
this.
G
G
A
A
This,
I
believe,
is
already
listed
as
a
blocker,
which
I
think
is
coming
down
to
the
version
of
yeah
that
if
we
use
those
the
particular
images
that
I
built
that
they
pre-start
docker-
and
we
don't
want
to
do
that
this
one.
I
think
that.
A
Perfect
dot
refresh
this
one
should
not
be
in
119.,
I'm
actually
working
on
this
one
right
now,
but
I'll
take
this
one
out
right
now
as
well.
I
think
I'm
not
logged
in
here
yeah!
Thank
you
and
this
one
I
think
as
well
deserves
to
should
be
in,
could
come
out.
I
feel
like
you've,
already
done
that,
so
I
think
it's
probably
obsolete,
but
let's
at
least
take
out
the
milestone
I'll
make
sure.
A
A
Case,
okay,
so
I
think
that's
that's
yeah.
It
sounds
like
there's
a
couple
of
issues
that
are
that
we
know
about,
but
otherwise
it
sounds
like
and
they're
mostly
my
mind
to
fix.
Otherwise
it
sounds
like
we're
in
good
shape,
for
119.
A
But
it's
not
feature
flagged
right,
so
it's
you,
but
you
have
to
opt
in
by
choosing
an
image
and
write
the
instance
and
the
instance
correct.
Sorry,
yes,
thank
you,
but
yes,
I
think
that's
a
great
one
to
to
tell
people
about
as
a
like.
Please
try
it
if
you're
interested
and
let
us
know,
but
we
do.
We
do
know
that
there
are
failures
currently
so
like
the
csi
failure
and.
A
A
Then,
if
we
can
burn
these
down,
then
we
can,
I
don't
think,
there's
any
reason
to
rush
a
release
of
it
right
now.
I
don't
know
if
anyone
disagrees,
but
we
can
probably
confirm
the
release
aim
to
confirm
there
is
on
the
fifth.
Does
that
make
sense?
Is
it
the
fifth.
B
I
was
thinking
that
maybe
we
do
a
release
so
from
those
issues
that
are
remaining
the
prefix
should
be
something
pretty
easy
to
merge.
So
it's
it's,
never
gonna
get
better
than
this.
B
A
H
A
B
C
A
B
Yeah
sounds
sounds
good
to
me.
Should
I
try
to
create
sorry,
I
think
I'm
ahead
of
things.
Should
I
create
a
120
alpha
one.
A
Yeah,
the
only
thing
we
have
in
the
120
alpha
list
is
getting
in
azure
support.
I
feel
like.
I
can
also
take
a
look
at
that.
I
don't
know
how
other
people
feel
like
whether
we
should
try
to
merge
that
and
do
the
alpha.
H
Great
then
I
I
I
propose
to
alpha
just
so
that
one
users
can
start
using
azure
and
test
with
it
and
also
to
get
our
version
on
the
master
branch
to
be.
A
B
But
it's
not
merged,
so
you
said
peter.
You
said
it's
over.
A
You
feel
free
to
approve
it.
I
will.
I
will
also
take
a
look
at
it,
but
I
I
I'm
sure
it's
fine,
like
I
think
I
looked
at
it
before
and
it
was
mostly
additive,
so
I
think
that's
the
that's
always.
My
bigger
concern
is
whether
it's,
whether
it's
gonna,
like
impact
existing
clusters,.
A
All
right
so
we'll
cut
some
stage
cut
a
a
alpha,
120
alpha
and
I
don't
know
what
separation
did
you
did
you
want
to
do
that
or
someone
else
want
to
volunteer.
I.
A
All
right,
that's
the
end
of
our
agenda.
I
don't
know
if
anyone
else
wants
to
bring
up
anything
in
the
last
minutes
that
we
have.
Otherwise
we
can.
B
I
think
there
are
a
few
gcp
issues.
I
don't
have
them
on
hand,
but
maybe
you
justin
or
eric
could
take
a
look
at
those
one
is
related
to
warnings
or
something
during
updates.
I
think
peter
took
a
look
at
it
a
while
ago,
but
couldn't
get
it
moved
and
I
don't
know.
Maybe
there
are
a
few
more,
maybe
something
simple
or
you
know
so
that
we
can
fix
some
bugs
before
the
release.
A
Yeah,
I
can
I'll
definitely
have
a
look
at
those
I
want
to.
I
want
to
add
the
authentication
support
that
is
currently
on
the
influence
of
aws.
Add
that
to
gcp
as
well.
So
I
can
also
take
a
look
at
that.
E
A
Well,
I
wish
everyone
a
very
happy
christmas
new
year
hanukkah,
however,
you
are
celebrating
it
and
we
will
see
you
all
in
the
new
year
or
on
the
fifth
or
on
the
15th.
Fifth
is
informal.
18Th
will
be
our
normal
bi-weekly
meeting.
Thank
you
all
for
a
wonderful
2020.