►
From YouTube: Kubernetes kops office hours 20191025
Description
Recording of the kops office hours meeting held on 20191025
A
Good
morning
afternoon,
everybody
it
is
October
25th
2019.
This
is
cops
office
hours.
I,
am
your
moderator
facilitator,
just
in
Santa
Barbara
I
work
at
Google.
This
remind
of
this
meeting
is
being
recorded
and
will
be
put
on
the
internet.
So
please
be
mindful
of
our
code
of
conduct,
which
boils
down
to
please
be
a
good
person.
I
have
pasted
a
link
to
the
agenda
in
the
chat
and
please
do
feel
free
to
put
your
name
on
there.
If
you
would
like
to,
if
you
have
other
items
to
put
on
the
agenda,
please
add
them.
A
We'll
try
to
figure
out
the
document.
Permissions
I
apologize,
everyone
that
is
having
to
add
suggestions
the
we.
We
have
a
couple
things
on
the
agenda,
three
things
on
the
agenda
and
then
we
can
go
through
our
release,
plan
and
update,
and
otherwise,
so
why
don't
we
just
jump
right
into
it.
Mike.
Do
you
want
to
talk
about
calico
or
yeah
totally.
B
B
B
You
notice
that
instances
can
no
longer
communicate
so
I'm,
not
I,
didn't
trying
to
keep
an
eye
on
it,
especially
since
we
use
calico
very
heavily
and
I
made
well
guy
made
a
bunch
of
changes
to
calico
and
then
I
think
I
made
a
bunch
of
changes
to
calico
to
kind
of,
like
you
know,
get
it
up
to
date
and
get
it
to
I,
try
to
reformat
it.
So
it
looks
more
like
what
calico
releases
I
went
back
three
to
try
to
confirm
that
you
know
we
didn't
make
any
drastic
changes
from
that.
B
But
yes,
so
so
it's
something
along
those
lines
now
he
said
it
worked
fine
in
the
cops
15
alpha,
I,
believe
and
then
once
he
went
to
the
beta.
That's
when
you
started
running
into
the
problems
and
the
beta
is
where
the
bigger
change
moving
from
versions.
I
believe
happened.
So
that's
just
some
background
on
it.
B
A
B
A
B
B
A
One
in
for
him
yeah
that
definitely
I,
think
I.
Think
the
only
I
think
this
one
was
somewhere
like
I
asked
whether
it
needs
to
go
into
a
114
or
115,
and
he
was
like
no
and
so
I
sort
of
let
it
slip
off
my
radar.
But
yes,
we
need
to
let's
get
into
116
alpha,
that's
cool,
hopefully
that
conflict
is
not
too
bad
I.
So
many
dogs
conflicts.
A
A
A
Yeah
I
don't
feel
like
there
is
an
underlying
issue
in
kubernetes,
which
we
probably
should
someone
should
fix.
But
yes,
it
feels
like
it's
yeah
I
think
that
seems
like
a
reasonable
thing
to
do.
One
fourteen
one
particular
we're
not
going
to
115
zero
I.
Think
that's
a
good
idea
and
then
yeah
I,
don't
know.
There's
anything
else.
Are
there
any
other
Pearson
should
I
open
this
cherry-pick
spreadsheet,
Jerry
I
think
that
was
the
only
one
since,
though,
that's
good
right,
that's
nice
and.
A
C
A
I
don't
know
we
could
change
the
color
of
that
I.
Don't
like
introduce
more
columns.
If
you
like
I,
don't
know
we
would
never
cherry
pick
something
back
to
114
zero
that
we
didn't
cherry
pick
to
114
one.
So
like
that's,
why
I?
Don't
you
know
I
mean
yeah,
we're
not
gonna
bifurcate,
that
that
would
be
yeah.
D
D
A
A
A
A
Ok
and
you
put
some
great
notes
in
there.
Thank
you
and
now
our
periodic
jobs.
Yes
constant
battle.
Yes,.
D
I
was
trying
to
figure
out
what's
causing
the
failures
in
test
grid
and
the
first
EBS
volumes
missing:
grenades
cluster
tech
causing
permissions
issue.
If
you
look
at
the
message
that
failure
messages,
they
say
unauthorized
operation
you're
not
authorized
to
perform
this
operation,
which
is
trying
to
attach-
and
you
yes,
valium
to
an
instance.
D
You
look
at
the
I-
am
policy
that
cops
creates.
It
only
allows
attach
volume
on
volumes
tagged
with
kubernetes
cluster,
so
that
was
my
first
guess
of
what
the
problem
might
be.
I
talked
to
Justin
briefly
about.
It
mentioned
this
in
testing
for
a
lower
cost
that
tried
to
fix
it,
but
it
looked
like
that
only
affects
GCE
a
few
lines
above
that
there's
a
check
for
if
we're
on,
GCE
preferred
not
on
GCE
and
returning
with
Ingo
script,
tester,
so
I
followed
that
rabbit
hole
and
cut
those
other
links,
but
I
think.
D
A
A
Don't
think
that
I
thought
the
chinko
script
tester
was
calling
it
directly.
Oh
sorry,
Jenko
script,
Chester
zone
which
calls
the
shell
one,
and
then
we
have
a
different
one,
which
does
its
direct
right
and
I
think
we
could
try
going
direct
the
reason
we
did
direct
only
for
GCE.
Originally,
it
was
because
it
was
lower
risk.
A
So
if
we
broke
it,
we'd
only
break
a
fraction
of
our
tests
and
not
100%
we're
just
given
the
eight
of
us
tests
are
broken
because
of
this
issue
were
not
passing
like
they
are
consistently
failing,
but
they
are
like
that.
Because
of
this
issue,
we
could
probably
try
the
direct
rather
than
exciting,
there's
an
effort
to
get
rid
of
that
Junko
e2e
script.
A
A
A
There's
no
easy
way
to
test
so
trivial
way
to
test
these
things
and
wow.
That's
not
too
bad,
but
there's
it's
pretty.
It's
a
little
tricky
to
test
these
things.
So
it's
often
a
case
of
just
try
it
and
see.
It
is
possible
to
convert
a
proud
test
job
into
a
local
doctor,
Joe,
which
you
or
don't
luck
with
local
docker
container,
which
you
can
then
run
and
pass
the
Christ
same
options
in,
and
it
should
more
or
less
work.
But
it's
probably
easier
just
to
you.
Try
it.
A
A
E
Just
saying
I'm
just
curious,
like
about
16,
seems
you
mentioned
in
the
release
plan.
Well,
but
from
my
own
access,
it
seems
from
massive
branches,
did
have
some
issues
so
I'm
learning.
What
some
apt
items
gonna
tell
me
about
this.
A
A
A
The
process
is
actually
getting
pretty
relatively
straightforward
now,
so
it
used
to
be
a
complicated
like
procedure
involving
like
if
you
have
to
push
to
some
stuff
sucker-
and
you
had
to
do
some
other
stuff-
it's
now
pretty
simple
and
that
you
just
have
to
upload
to
either
a
Google
Cloud
storage
or
an
s3
bucket,
and
it's
should
all
be
done
for
you.
So
I
will
do
that
doc,
but
I
think
I
think
that
the
I
think
that
will
fix
it
and
I
think
the
I
always
put
a
little
bread
crumb.
A
If
you
do
want
to
look
for
it,
which
is
like
search
for
upload
test,
that's
an
sp2
document
document
properly
to
upload,
but
also
that's
so.
The
reason
I
believe.
The
reason
why
it
might
not
work
is
because
it's
still
pointing
at
the
115
0
node
up
and
I'm
guessing
there
is
some
incompatibility,
so
that
will
get
fixed
when
I
do
the
116
0
upload,
which
I
probably
should
have.
We
forgot
about
you
that
anyway,
like
maybe,
we
always
cut
alpha
1
I,
where
I
will
do
the.
A
E
A
E
A
B
I
dropped
that
in
there
I
think
that's
what
you're
looking
for.
You
can
use
a
combination
of
that
there's
also
a
command
with
basil.
That's
a
little
more
straightforward,
using
basil
upload,
if
you,
if
you
use
basil,
but
look
through
that
directory
the
development
directory
and
there's
all
of
those
that's
what
we're
working
to
condense
now
actually
is
one
of
the
the
next
topic
I've
drawn
there
real,
quick.
It's
some.
A
B
B
So
so
we
for
those
that
weren't
aware
we
basically
had
an
outstanding
PR
to
get
a
bunch
of
the
changes
I
made
in
I
had
that
held
because
I
figured
there
would
be
blockers
from
changes
that
were
made
to
Doc's,
but
there
weren't,
so
it
was
ready
to
merge
so
I
merged
it
in
and
then
I'll
make
some
follow
up
smaller
PRS
because
it
was
just
getting
too
big.
I.
B
B
B
B
A
A
D
Yes,
I
noticed
we
have
1806
I,
think
we
have
1809.
For
me
just
stretch
kubernetes
114
I
believe
is
the
first
that
is
officially
supports.
1809
I
was
looking
at
updating
other
distributions
to
1809
and
notice.
It
doesn't
support
Jessie
I'm
wondering
how
we
handle
that,
given
that
the
docker
version
is
a
property
that
cluster
manifests,
but
the
distribution
specific
issue.
A
A
A
And
actually
I'd,
like
I'd,
prefer
us
to
get
to
that
model
in
general,
rather
than
relying
on
the
relying
on
the
OS
packaging.
Because
of
exactly
these
challenges,
particularly
around
security
vulnerabilities
I,
don't
want
to
go
and
like
rip
out
all
the
existing
versions,
but
like
gradually,
like
with
container
D
I,
think
we
should
start
with
a
tar.gz
model
and
see
how
it
goes
rather
than
hoping
that
the
the
OS
package
or
the
Technica
darker
packages
match
what
we
actually
want.
D
A
A
A
B
Yeah,
so
this
is
I,
someone
posted
this
in
slack
earlier.
It's
actually
similar
to
what
we
were
just
talking
about
with
docker
stuff
I
thought
it
might
be
worthwhile
bringing
up
right
now.
I
haven't
commented
on
it
because
I
don't
really
I'm,
not
sure
what
the
right
thing
to
do
is
whether
we
should
allow
sorry,
my
computer,
still
kitchen
Wow.
You
know
just
basically
inline
overrides
of
the
docker
package.
B
I
mean
part
of
me
feels
like
it's
a
pretty
logical
approach
of
if
the
user
really
wants
to
go
all
in
and
do
this
themselves
great,
but
at
the
same
time
you
know
it
kind
of
breaks
some
of
our
patterns,
so
I
just
figured
some
highs
would
be
helpful.
If
anyone
else
has
views
on
it,
it
seems
to
be
a
appear
with
a
few
people
interested
in
it.
A
It
it
doesn't
seem
unreasonable,
so
we
have
the
example.
I
would
give
would
be
the
kubernetes
version
yeah
like
where
we,
you
know
you,
you
can
specify
the
one
1704
one
or
you
can
specify
a
URL,
and
we
use
that
as
the
base
URL,
that
we
have
some
limitations
about
only
expect
the
structure
of
that
repo
to
look
like,
but
it
works,
and
we
did
that
for
the
purpose
of
testing
of
CI.
A
The
original
motivation
was
for
CI,
but
doesn't
seem
like
we
should
not
doesn't
seem
like.
We
should
disallow
this,
particularly
if
it's
a
tar.gz
I,
don't
know,
but
whether
this
was
actually
supposed
to
be
about.
Was
this
about
tar.gz
or
was
this
about
yeah
anyway?
I'll
have
a
look
I,
don't
they're,
always
gonna,
be
some
okay,
that's
a
fault!
It's
a
pretty
heavy
specification,
all
right,
yeah.
B
A
Yes,
it's
a
good
one
to
think
about.
Yes,
it
was,
it
was
less
so.
The
structure
mirrors
the
structure
in
the
doc
Reducto
format,
which
effectively
is
like
changing
an
implementation
detail
into
a
contract
with
the
user
which
I'm
not
wild
about
because
they
all
that
that
format
is
not
intended.
Like
has
not
been
designed
for
like
public
consumption
as
it
were.
It's
like
implementation
detail.
A
A
A
A
Alright,
let's
see
okay,
so
we
got
114
0
beta,
1
out
the
door
I
think
10
days
ago,
so
early
last
week,
I
did
not
I
did
not
to
get
116
0
alpha
1
other
row,
but
I
think
we're
close.
Now
there
was
there
were
a
couple
of
PRS
that
we
needed
to
get
in
the
member
list,
PR
the
something
else:
something
OpenStack
some
on
OpenStack
PR.
That
was
important.
That
I
can't
remember
the
details
up
and
a
couple
controller
support,
OpenStack
and
then
also
there's
a
sed
manager.
A
Support
for
digital
ocean,
which
we
got
in
this
morning
into
a
city,
manager
and
I
think
may
or
may
not
be
merging
soon
in
two
cops
itself.
So
there's
some
test
issues,
but
other
than
that
we
have
1
PR,
which
we've
identified
that
we
should
get
in,
which
is
the
jessee's
PR
around
over
using
existing
subnets
and
networks.
We
have
a
note
oops.
A
A
A
Yes,
we
should
do
that.
Yeah
I
think
that's
fine,
okay
and
then
we
some
releases
that
we
talked
about
on
sort
of
the
stable.
Oh,
oh
yeah,
so
we've
decided
that
we're
not
gonna
do
115
zero.
Not
let's
leave
that
there
so
don't
accidentally.
Do
it
115
see
we're
not
gonna,
do
a
14-0
in
the
next
two
weeks
or
it's
unlikely
that
we
will,
because
there
are
two
fairly
substantial
blockers
with
on
CNI
via
CBC,
the
AWP,
CC
and
I.
A
A
I
guess
is
the
question:
does
anyone
know
of
any
other
issues
or
first-time
prefer
to
let
it
bake
if
we
get
those
two
I
prefer
to
do
a
beta
2,
but
we
can
see
we
can
see
how
how
how
how
much
change
is
in
the
two
fixes
and
then
one
fort
we're
gonna
do
a
cups,
114
one
which
has
a
cherry
pick
of
the
fix
for
resource
version,
core
DMS,
the
coordinates,
research
version,
resource
version,
as
Brian
pointed
out.
Thank
you
right
and
then
we
have
general
ami
promotion
and
kubernetes
version
promotion.
A
A
D
A
The
process
is
remarkably,
not
exciting,
but
yes,
actually,
yes,
I'm
happy
to
do
that.
B.
If
you
are
interested
in
how
it
happens,
these
there
is
a
document
that
is
in
like
kubernetes,
cops,
docks
development
release,
I
think
they
said,
docks,
release
yep
and
you
can
sort
of
see
the
process
and
it's
not
too
bad.
A
A
A
A
Container
and
I
am
personally
adding
supporter.
I
am
working
on
support
for
at
a
promotion
of
non
container
non
image,
artifacts,
which
we
have
more
of,
and
if
we
get
that
going,
that
I
think
will
be
in
great
shape.
We
do
not
yet
have
a
solution
for
github,
so
I
don't
but
I
can't
imagine.
That
would
be
too
bad
other
than
the
fact
that
it's
sort
of
simplified
right
now,
because
it's
all
on
Google
infrastructure,
so
it
all
uses
Google
OAuth
words
now
we'd
have
to
hold
a
github
credential.
Yes,.
B
A
It's
still
it
is,
it
is
still
on
the
additions.
There's
this
group
that
is
trying
to
build
infrastructure
that
is
owned
and
managed
by
the
CN
CF
and
is
not
only
managed
by
Googlers,
and
although
currently
it
is
using
a
lot
of
Google
cloud
platform
infrastructure,
it
is
not
like
otherwise
tied
to
Google,
and
the
hope
is
that,
like
there
can
be
support
for
other
or
there
can
be
other
cloud
services
that
are
also
supported,
be
a
github
or
AWS,
or
you
know,
or
whatever
or
OpenStack
or
whatever
it
might
be.
A
If
we
yes,
but
anyway,
we're
starting
simple
like
it's
always,
the
battle
is
to
start
simple.
So
first
step
is
get
these
promotions
going.
I.
Think,
although
we
are,
although
we
are
mostly
concerned
about
non
image,
artifacts
I
think
I
started
with
just
pushing
the
images
in
this
PR
I,
don't
know
if
I've
since
ripped
out
all
the
images
that
might
be
the
comment:
yeah.
Okay,
no,
it's
not!
Okay,
yes,
but
yes,
we're
gradually
riffing
we're
gradually
reducing
number
of
images.
A
We
actually
need
to
push
so
I
want
to
get
this
in
before
we
before
we
get
rid
of
the
last
one
I
guess.
Otherwise
it
will
be
a
little
academic,
but
but
yes,
starting
with
images
that
we
can
push
the
binary
artifacts
into
a
GCS
bucket
and
then
we
can
promote
them
from
there
and
we'll
have
to
like
figure
out
how
we
get
them
promoted
from
the
staging
into
a
production
body,
but
I
think
we
actually
have.
A
We
have
artifacts
of
kate's
Tadeo,
which
is
currently
in
one
hour
in
our
alias
list,
and
there
is
a
slightly
more
manual
promotion
process
right
now
that
runs
the
non
non
image.
Promoter
and
I
am
promoting
occasionally
some
of
our
artifacts,
so
I
think
with
116
zero
alpha
one,
it
will
actually
land
and
then
we
might
actually
start
pulling
from
there.
The
nice
thing
about
the
we
have
this
like
resolution
process,
where
we
try
a
bunch
of
mirrors
and
the
three
mirrors
we
will
have
will
be
the
existing
s3
bucket,
the
artifact
skates
I/o.
B
It's
a
fairly
diverse
set
of
providers.
There
that's
awesome.
This
is
super
exciting
to
me
and
I'm.
Looking
forward
to
the
day
when,
like
we
don't
have
to
ask
you
know,
did
you
try
proto
cube?
You
know,
you
know
copy
a
protic,
you
in
Cuba,
you
know,
and
and
now
we
can
just
say
actually
just
run
this
command
and
we'll
pull
the
latest.
You
know
build
off
master
and
that
would
be
really
cool.
Yes,.