►
From YouTube: Kubernetes kops office hours 20200103
Description
Recording of the kops office hours meeting held on 20200103
A
Hello,
everybody.
This
is
cups
office
hours
today
is
Friday
January
3rd
2020,
happy
New,
Year
to
everybody.
We
hopefully
will
have.
We
know
we're.
Gonna
have
a
great
year
because
of
all
the
exciting
things
are
already
in
the
PR
backlog
and
let's
we
have
an
agenda
for
today.
So
a
reminder,
this
meeting
is
being
recorded
and
will
be
put
on
the
internet
to
be
mindful,
therefore,
of
our
code
of
conduct,
which
boils
down
to
please
be
a
big
person
and
be
respectful
of
others.
A
B
A
I
think
it's
been
a
while
we
were
talking
about
doing
them
more
regularly
and
yes,
it's
certainly
something
I
would
I
will
both
do
a
new
set
of
am
eyes,
try
to
get
in
the
habit
of
it
and
see
where
we
are
with
the
automated
or
the
image
builder
project,
which
is
trying
to
bring
this
into
more
into
the
fold,
hopefully
getting
it
to
the
point
where
it
can
be
more
I.
Guess
automated
yeah.
A
C
A
A
Registries,
I,
don't
know
what
they're
called,
but
the
docker
apt
apt
sources
was
down.
The
image
would
still
come
up
because
it
has
darker
pre-installed,
typically
or
the
intent
was
that
it
was
only
an
optimization
so
that
you
can
use
any
image.
You
could
use
a
bare
Debian
image.
You
can
use
a
completely
different
kernel
or
many
different
OS
and
it
would
work
just
if
you
happen
to
be
using
a
Nami
which
happened
to
have
the
correct
version
of
docker
pre-installed.
A
Then
it
would
just
go
faster
and
not
have
the
external
that
needs
to
download
docker.
We
haven't
historically
installed
kubernetes
versions
so
cubelet,
primarily
because
we
make
a
Nami
for
each
minor
version
and
not
for
each
patch
version
of
of
communities.
That
is
something
we'd
look
at
changing,
but
it
isn't
something
we
have
changed
to
date.
I
know,
we've
had
some
issues
here
and
that
we
don't
necessarily
correctly
handle
like
removal
of
docker
versions.
If
I
recall
correctly,
I'm,
not
sure
exactly
that's
true
or
not,
but
it
sounds
like.
A
We
also
have
an
issue
here
where
we,
when
people
use
in
container
D,
we
obviously
don't
want
darker
pre-installed
I
think,
but
we
need
to
make
sure
is
I
think
that
we
need
to
make
sure
that
the
the
installation
is
only
an
optimization.
So,
in
other
words
docker
is
shouldn't
be
running
I
guess,
but
you
don't
mean
that
pre-installed
yeah.
A
A
C
Underst
and
I
I
did
some
work
on
the
code
of
the
packages.
The
problem
with
this
optimization
is
that,
at
least
in
the
package
install
code
is
that
you
don't
get
the
new
versions
of
any
package.
So
basically,
if
you
install
some
newer,
docker
or
whatever,
or
when
you
bring
up
the
VM,
you
just
get
watching
the
VM
pre-installed.
So
if
there
is
a
security
update
or
something
you
get
it
during
the
night,
instead
of
getting
it
right
away,
doing.
A
Yes,
so
it's
it's
a
little
we're
a
little
weird
here,
and
so
the
order
of
upgrades
are
on
I
think
by
default.
But
if
you
want
to
be
more
in
theory,
if
you
want
to
be
all
like
declarative
and
fully
specified,
you
in
theory
should
turn
off
auto
upgrades
and
manage
the
docker
installation
yourself,
like
that
version
yourself.
That
doesn't
really
make
a
ton
of
sense
for
security
updates.
We're
probably
you
actually
want
the
update.
So
it's
a
weird
like
in
between
space,
where
we
do
well.
We
do
have
auto
upgrades
turned
on
by
default.
A
C
That's
a
different,
totally
different
thing,
because
in
a
bigger
company
you
would
have
internal
servers
where
your
update,
our
updates,
are
pulled,
and
you
anyway,
get
auto
updates.
What
from
those
that
are
certified
or
whatever
the
IT
department
does?
Yes,
I,
don't
think
anyone
wants
to
run
I,
don't
know
ansible
or
something
to
update
all
the
packages
every
week.
A
C
A
A
A
C
A
C
Anyway,
I'll
try
to
make
something
around
it.
I
made
something
right
now
we
install
things
using
yum
and
apt,
not
dpkg
and
RPM
anymore,
okay,
so
this
at
least
pulls
the
dependencies.
So
if
there
is
a
dependency
with
like
it
was
for
CentOS,
we
couldn't
pull
the
right
dependency
because
it
was
hard-coded
in
my
dependency
in
extra
packages.
Right
now
we
pull
and
we
pull
everything
that's
latest
required
by
a
docker.
C
A
That
sounds
yeah
that
sounds
great
I,
mean
I,
think
I
think
we.
We
definitely
need
to
get
the
get
it's
of
it.
For
example,
if
container
D
is,
if
you're
using
it
in
early
that
we
aren't
running
darker,
because
you
happen
to
be
using
this
install
right.
So
that's,
it
seems
like
a
fairly
straight
or
a
fairly
important
one
to
do
first
and
then,
yes,
any
would.
A
Yeah,
okay,
which
is
that
the
exact
miner
is
from
var.
We
should
move
that
or
well.
What
are
we
gonna
do
about
the
different
OSS
in
their
ability
to
Joanna?
Have
you
want
to
explain
sorry
mm
pretty.
C
A
C
A
Makes
no
sense
he
art
out
cops
been
enough.
Criminal
he's
been
makes
a
lot
of
sense
to
me
an
obscene
I
bet,
I
guess
I,
don't
know
if
anyone
else
has
any
thoughts
on
that
we
always
have
the
document
in
the
release.
Notes
that
we
are,
we
are,
as
of
one
1917
I
remember
we
are
going
to
put
the
binaries
into
a
particular
location
or
a
different
locations.
A
Yes,
we
can
so
I
think
we
should
move
it
every.
Oh,
we
should
move
all
OS
is
to
opt
cops
baton,
for
example,
at
a
particular
version
like
118
or
117.
Whatever
we
choose,
and
then
we
should
move
whatever
version
we
choose
to
back
port
as
it
were
real
or
cherry-pick
back
or
l8
support.
We
should
like
say
like
that.
The
path
for
rl8
should
be
the
future
path.
The
OP
cop
spin
mm-hmm.
A
So
then
we
and
we're
not
sort
of
not
constrained.
But
yes,
if
we,
if
we
sort
of
know
what
we're
gonna
put
it,
we
should,
as
we
add,
support
for
Noah's
new
OSS.
We
should
put
them
in
the
future
location.
Oh
yeah,
I
think
that's
great
I
will
have
a
look
at
the
the
PR
81
to
which
I'm
guessing
is
moving.
It
and
I
think
yeah
up
up
to
me.
It
feels
like
a
great
place
for
them.
It's.
A
C
A
So
we
we
have
these
old.
The
tag
thing
is
this
old
mechanism
that
we're
trying
to
get
rid
of
and
I
think
this
is
one
of
the
last
things
and
like
gradually
I
do
when
I
hit
something
that
uses
tags,
move
it
to
not
use
tags.
So
there's
that
one
bit
which
is
like
we
have
I,
think
a
handful
of
things
left
that
use
that
tags
mechanism
which
we're
trying
to
get
rid
of,
but
that's
one
thing
which
is
moving
it
from
tags
to
like
a
more
go
version.
A
Originally,
we
we
had
a
I
guess
something
more
like
ansible
or,
like
you
know,
a
directory
structure
based
approach,
and
it
turned
out
that
go
code
was
much
better
and
easier
to
understand,
particularly
as
things
got
more
complicated.
So
that's
why
we
moved
away
from
tags
now
why
we
have
this
health
check
thing
at
all.
A
A
Know
whether
we
really
still
need
it,
though,
like
there's
this
in
fear,
it
shouldn't
be
necessary.
In
theory
like
docker
shouldn't,
hang
and
container,
they
shouldn't
hang
in
practice.
We
might
still
need
it
like
occasionally
we
see,
but
but
on
the
other
hand
like
other
OSS,
don't
have
it
and
they
seem
to
get
away
with
it.
So.
C
A
C
A
C
But
it's
also
bad
because,
let's
say
I
didn't
know
about
it
and
I'm
using
cops
for
I,
don't
know
more
than
a
year,
I
would
say
close
to
2
and
didn't
know
about
it.
So
could
hide
the
fact
that
your
daughter
is
unstable.
You
don't
know
any
you
see
it
that
it's
still
running
when
you
try
to
debug,
but
you
may
not
notice
that
it
was
restarted
at
some
point
by
the
script.
C
A
That's
very
that's
very
fair
in
a
good
point:
yes,
III,
guess,
I
guess
a
flag
or
field
would
be
a
great
way
to
expose
that
and
sort
of
the
same
argument
that
you
persuaded
me
on
with
like
the
idea
of
exposing
the
fact
that
docker
uses
container
D.
Now
to
me,
it
feels
like
a
good
idea.
I
think
that
would
be
good
I'll,
also
look
at
whether
we
like
I'll
try
to
find
out
whether
it's
still
needed.
If
that
was
beautiful,.
C
C
A
F
A
Think
it's
a
I
can
have
a
look.
It's
a
good
idea.
It's
a
good
idea,
the
only
my
only
he
liked
worry
would
be
like
if
it
is
a
container
that
itself
is
checking
it.
You
know
it's
checking
itself,
but
I.
Think
yeah
no
permit
vector
would
be
a
great
way
to
do
that
as
well.
We
can
also
express
it
as
an
add-on,
even
if
it
isn't
a
even
if
it
isn't
a
pod
effectively,
but
I
would
rather
let's
let's
follow
that
line
of
pursuit
as
one
of
the
options
that
we're
looking
at.
E
A
D
A
This
has
been
around
for
a
very
long
time
and
it
used
to
be
much
much
more
of
a
problem
and
it
used
to
be
like
you
would
very
rapidly
know
if
you
weren't
running
it
now,
I
think
a
lot
of
OSS
aren't
running
in
our
fine
but
yeah.
That's,
let's
take
it
on
to
the
ER
and
discuss
these
options.
I
think
that's
probably
the
way
to
go
and
I
will
try
to
find
out
from
the
no
team
what
state
of
play
is.
B
B
C
And
it
was
a
and
fresh
empty
I
mean
nothing
configured
cluster.
Yes,.
A
Can
have
a
look
at
the
let's
assume
if
it
if
it,
let's
assume
that
it
is
broken,
not
saying
you're
wrong
job,
but
that
the
like,
if
there,
if
it
if
it
isn't
broken,
there's
no
discussion
as
far
as
I
can
tell
so.
Let's
assume
it
is
broken
and
then
well.
It's
definitely
broken
for
one
night
and
earlier:
okay,
okay,
so
yeah
there's
some
number
there's
some
number
where
it
is
broken,
I'm
one
night,
but
one
line
was
where
we
gonna
was
our
deprecation
threshold
anyway,
I
believe
when.
A
Forth,
okay,
okay,
so
there
is
some
number
some
version
newer
than
our
fan:
deprecation
version.
Where
we
have
broken
things,
I
feel
like
we,
we
probably
should.
Okay
well
I
mean
deprecations
deprecation,
so
maybe
we
shouldn't
just
fix
it.
It
doesn't
look
that
bad,
a
patch,
I
I,
don't
know
what
there
were.
There
were
people
that
were
using
older
versions,
I
think
right.
How
would
does
anyone
want
to
think
we
should
continue
to
support
these
versions?
I
guess
I!
Guess.
A
A
You
know
which
is
like
we
know
we
want
to
get
something
out
there.
We
would
and
we
want
to
accelerate
it
like.
We
want
to
be
nice
to
people
and
tell
them
that
we
are
going
to
like
narrow
the
window
overtime,
probably
down
to
whatever
we
said
I
can't
we
said
six
or
eight,
but
like
some
number
of
that
of
that
nature,
I
can
have
a
look.
A
B
A
Think
that
that's,
if
we're
gonna,
have
a
deprecation
policy,
we're
gonna
formally
say
yes,
we,
like
you,
would
do
support
these
versions.
We
should
have
the
test.
That
means
we
have
some
notion
of
whether
or
not
we've
broken
them.
So
we
don't
have
to
sort
of
have
the
debate
is
one
nine
is
110,
is
111,
broken
yeah
I.
Think
one.
B
A
A
C
A
A
C
A
Yeah
I
agree
I,
just
I,
guess
like
we
try
not
to
introduce
changes
into
previous
versions
or
something
somewhere
went
awry
there
and
I
will
try
to
find
out
what
that
is
and
just
free
time.
Do
it
again
right,
yeah,
John,
I,
see
I,
see
you
smiling
because
you
are.
You
are
proven
correct
that
the
10
release
policy
is
indeed
problematic.
I.
B
A
A
I
I'm
here
sorry
yeah,
so
I
am
I
completely
broke.
These
tests.
I
guess
yesterday
has
something
to
do
with
the
HCL
and
I'm
not
really
sure,
what's
actually
making
them
break,
because
it's
saying
that
it's
breaking
on
a
token,
that's
required
to
be
there.
It's
the
role.
So
can
so
I'm
not
really
sure
yeah
not
really
sure
why
they're
broken
I
think
I
may
be
thinking
about
how
the
tests
are
run,
mm-hmm
and
maybe
a
wrong
way,
but
I
don't
know
I.
A
C
A
A
A
We
hope
to
get
more
stuff
out
of
Travis
into
prowl
jobs,
okay,
but
the
idea
is
that
if
you
run
make
CI,
you
should
run
the
complete
set
of
tests
and
Travis
CIA
just
happens
to
be
the
target,
that's
run
by
Travis.
So
if
you
wanted
to
recreate
what
Travis
run
you
could
what
Travis
runs
you
could
run
make
Travis
CI.
But
if
you
want
to
check
everything,
you
should
probably
run
make
CI.
Okay,.
J
H
K
A
K
So
we're
currently
debugging
a
problem
with
it
city
manager
or
with
our
cops
cluster
that
we've
built
having
problems
getting
all
three
HCD
members
up
and
running.
So
we
have
two
out
of
the
three
working
at
the
moment
and
just
hoping
nothing
goes
wrong
there.
It's
a
development
cluster,
so
it's
not
too
bad,
but
more.
K
It's
then
tries
to
gossip
and
find
the
other
servers
before
actually
starting
it
CD
itself
Corral,
and
that
was
that
was
one
of
the
things
that
wasn't
immediately
clear
is
that
it
doesn't
immediately
start
out
today
and
the
problem
that
we're
facing
is
that
on
the
the
node
that
keeps
coming
up
as
unhealthy.
Is
that
it
doesn't?
It
never
actually
starts
at
City.
K
So
all
the
other
members
try
and
communicate
with
it
and
say
it's
unhealthy
because
it
CD
can't
be
communicated
with
from
their
port,
and
the
reason
is
that,
if
you
actually
look
at
that
one,
it's
saying
that
it's
I'm
not
exactly
sure.
Why,
but
like
something
in
the
initialization
process,
is
in
the
wrong
state,
and
it
maybe
thinks
that
it's
already
running
it
CD,
but
it's
not
so
I,
think
just
some
debugging
tips
or
or
how
to
sort
of
go
about
this
sort
of
thing.
I'm,
not
sure.
K
A
Is
I
think
the
it
would
be
super
helpful,
but
what
you
said
just
now
about
the
the
idea
that,
like
the
beginner's
mind
as
it
were,
to
capture
that,
if
you
could
send
a
doc
or
something
just
to
say,
like
you
know,
the
first
thing
it
does
is
it
tries
to
bring
up
the
control
layer
and
gossip
and
elect
a
leader,
and
then
it
starts
SUV.
That's
like
yeah,
just
something
that
I
hadn't
thought
right.
A
So
we
great
to
get,
if
you
don't
mind,
sending
any
duck
around
that
at
all
the
the
the
way
to
check
what's
going
on
here
is
the
machine
which
isn't
coming
up
to
see
or
is
not,
which
is
that
the
British,
the
leader,
can't
contact
basically
to
see
why
it
can't
I'm
not
sure
it's
the
same
machine
as
this
one,
but
to
when
SV
runs
it
will.
Its
logs
are
interspersed
into
the
SD
manager
logs,
which
is
sort
of
annoying
yes
yeah,
but.
K
C
K
A
E
D
A
K
K
A
K
We've
done
some
some
weird
things
to
this
cluster,
so
there
might
be
like
some
old
state
left
around
or
something
which
is
causing
this,
because
I
think
we
initially
created
it
without
a
load
balancer,
and
then
we
liked
or
some
stuff
down
and
then
rebuilt
it.
So
you
may
try
just
issuing
a
restore
yeah
that.
E
K
One
of
the
error
messages-
not
this
one
but
was
was
saying-
do
a
restore
from
backup
and
I
tried
that
it
City
Manager
restore
command
but
yeah.
It's
essentially
the
same
thing
and
and
also
some
documentation
on
how
that
restore
actually
works
would
be
because
it
did
seem
to
be
kind
of
magic.
I
thought.
A
K
A
H
A
K
A
A
It
should
gossip
around.
It
should
be
like
other
locations
to
check,
but
I
will?
Okay,
that's
as
long
as
the
current.
There
is
an
IP
for
to
see
which
it
expects.
It
says
somewhere
like
that
IP
address,
were
it
dialed
and
tried
to
fail
the
two
five
three
address
like
making
sure
that
is
actually
the
correct
address:
yeah,
okay,
cool,
okay,.
A
B
C
A
A
We
have
reached
the
end
of
our
agenda.
There
are
oh
yeah
release
plan
for
the
next
two
weeks.
Good
point
we
should
do
definitely
we
should
agree
their
release
time
for
the
next
two
weeks.
Thank
you.
Whoever's.
Readiness
should
bring
back
one.
Thank
you
new
a.m.
eyes
and
then
I
don't
know
how
people
feel
about
updating
or
118
zero.
Alpha
does
a
little.
We
should
let's
come.
C
It's
pretty
good
technology
preview
for
container
D
or
whatever
else
people
want
to
see.
For
me,
1.18
works
reasonably
in
tests,
I
mean
I,
don't
see
things
not
working,
so
we
could
start
with
all
for
now
and
once
it
is,
we
can
keep
up
with
the
official
kubernetes
when
they
release
the
better.
Also
in
two
weeks
or
something
is.
A
G
L
A
A
A
H
A
H
A
A
Ok,
oh
yeah,
and
then
we
can
do
an
alpha
2
of
117
catch
up
on
any
outstanding
things
and
hopefully
get
some
people,
maybe
trying
that
out
or
trying
it
out
that
sound
good
to
everybody
Wow
two
thumbs
up.
Thank
you.
Alright,
I,
don't
know
if
there's
anything
else.
That
feels
like
a
good
I
think.
The
other
thing
to
do
is
get
lots
of
like
get
that
yet
the
PRS
down
I
think
John.
We
hope
you'll
get
your
tainting
tank
before
role
I.
Think
we
I
like
the
idea
of
soft
painting.
A
I
A
C
C
A
C
A
A
You
can
watch
the
notes
and,
like
the
notes,
have
the
provider
ID
and
you
can
from
there
know
when
a
note
is
joined,
so
that's
it's
not
impossible
to
do
elsewhere.
I
do
know
that
there
are
other
other
companies
that
have
automated
systems
that
try
to
enforce
this
rule
as
well
so
I
and
will
delete
things
automatically.
For
example,
if
they
aren't
tagged,
I
don't
know.
If
anyone
here
has
this
works,
it's
actually
cut
place
or
has
an
existing
workaround
I'd
suggest.
L
It
is
confirmed
as
true
we
run
the
same
issue.
I
just
told
them
to
just
look
at
the
instances,
because
they're
all
tagged
properly,
you're
right,
I,
I,
think
there's
a
valid
use
case
for
getting
this
fixed,
but
I.
Think
Justin
is
right.
Let's
get
an
issue
open.
We
can
dump
some
ideas
there
and
kind
of
pick
up
path
forward.
Yeah.
E
E
F
A
J
J
C
A
We
we
have
the
controller
now
and
it
does
observe
nodes,
so
we
should
be,
and
it
has
I
think
it
WS
permissions,
or
it
has
some
ADA
videos
permissions
already
because
it
has
to
like
cross-check
instances,
although
yeah
no
I
think
it
does
I
think
it
was
all
that
I
think
all
those
things
are
true.
They
certainly
will
be
true
over
time,
so
it
wouldn't
be
too
hard
to
put
it
right.
There
I,
don't
know
whether
we
want
to
have
it
flagged
or
controlled
in
some
way
or
whether
we
should
just
start
doing
it.
A
B
B
A
B
B
You
say
that
again,
so
we
are
cubed.
I
am
set
up.
We
have
a.
We
have
namespace
enforcement's,
so
that
pods
in
a
certain
namespace
can
only
assume
certain
roles,
and
then
we
have
a
default
role
for
pods
that
don't
have
the
role
our
annotation
on
them
and
that
we
have
assigned
to
a
role
with
no
permissions
on
it,
and
so
it
sounds
like
with
our
current
set
up.
B
The
copter
controller
would
break
because
cubed
I
am
would
give
it
credentials
for
the
role
with
no
permissions,
rather
than
letting
it
pass
through
and
inherit
the
nodes
credentials
so
I
think
at
least
we
should
allow
the
cops
controller
to
have
annotations
added
to
it.
I'm,
not
sure,
if
that's
supportive
already,
but
we
should
make
that
configurable.
A
Or
explicitly
managers
role,
but
yes,
yeah
I,
think
that'd
be
good.
There's
there's
also
you
know,
there's
the
AWS
did
everyone
suppose
project,
which
is
the
nativist
identity
for.
B
Service
accounts
yeah.
Thank
you,
I'm,
working
on
adding
support
for
that
in
cops
too.
But
that's
I'm
not
far
along
enough
to
bring
it
up
here
and
there's
a
few
roadblocks
that
we're
gonna
need
to
discuss
so
I
think
it
like
publish
it
needs
to
publish
that
JW
Tina
needs
files
in
your
state
store
that
are
public,
read
access,
which
I
think
are,
as
three
layer
doesn't
support
easily.
A
The
s3
Wow
okay
on
GCS,
we
actually
started
doing
that.
So
GCS
has
different,
let's
just
say,
different
from
approach
to
permissions,
and
so
there
is
an
actual
abstraction
which
might
be
there.
I
we
should
certainly
talk
about
this
properly.
You
know
like
in
with
more
time,
but
there
is
an
act
abstraction
which
might
allow
it
I,
don't
know
where
there's
a
good
idea
to
have
individual
files
in
a
bucket
as
sensitive
as
the
cup
state
store,
which
are
public
or
whether
we
should
create
a
second
bucket
for
someone
give
it
I'll.
L
It
also
doesn't
work
if
you
run
your
own
OID
C
provider,
that's
private,
like
Dex,
or
something
like
that.
So
that's
an
Amazon
problem.
We
have
an
open
ticket
with
the
one
that
it
doesn't
mmm-hmm
yeah
career
day,
cops
and
kubernetes
work,
fine
with
it,
but
they
on
the
I
am
side
views
they
need
to
connect
to
it,
and
if
it's
private,
they
don't
have
the
networking
way
to
connect
to
it.
Okay,
anyway,.
A
Not
related
all
right,
yes,
it's
definitely
a
bigger
topic,
I
guess
for
a
future
time,
and
we
are,
we
are
at
time,
I,
don't
know
whether
anyone
had
in
the
last
items,
but
otherwise
this
is
a
great
start
to
2020.
I.
Think,
thank
you,
everyone
for
everything
you
are
doing
and
we
will
see
everyone
in
two
weeks.