►
From YouTube: Kubernetes kops office hours 20201106
Description
Recording of the kops office hours meeting held on 20201106
A
Hello,
everybody
and
welcome
to
the
chaos
office
hours
bi-weekly
meeting
today
is
friday
november
6
2020..
I
am
your
moderator,
facilitator,
justin,
santa
barbara.
I
work
at
google
a
reminder.
This
meeting
is
being
recorded
and
will
be
put
up
put
on
the
internet
uploaded
to
youtube
and
to
please
be
mindful
of
our
code
of
conduct,
we
do
have
an
agenda
whose
link-
I
am
pasting
now
into
our
chat.
A
There
are
a
couple
of
things
on
there
and
please
feel
free
to
add
other
things
and
feel
free
to
add
your
name
for
anyone
that
is
watching
the
video
and
wants
to
correlate
that
back
and
otherwise.
I
think
we
can
jump
straight
into
things.
A
The
we
had
a
couple
of
action
items
from
last
time
we
had
a
a
release
of
118.2
as
sort
of
a
patch
release
in
our
118
stable
series,
which
I
believe
I
did
and
we
had
a
plan
to
do:
119
0,
beta
1..
I
think
that
we
had
some
blockers
that
I
don't
think
we
fully
resolve,
or
we
had
some
things.
We
weren't
entirely
clear
on
whether
they
were
resolved
or
not.
A
B
C
Yes,
I
was
hoping
mostly
to
get
in
the
launch
template
changes
so
that
it's
part
of
the
beta
also
maybe
the
acm
patch-
that
peter
is
working
on
and
we
tried
it
together,
but
probably
we'll
have
more
time
during
the
weekend
or
something
so
to
get
it
at
least
more
stable
to
be
part
of
the
beta,
but
other
than
that.
I
don't
think
that
there
are
major
issues
there
so
next
week,
probably
I
can
do
the
beta
one.
A
I'll
be
great
yeah
I
mean
I
I'm
just
sorry
stammering
though
at
this
point
I
think
we're
only
reviewing
the
actual
items
but
yeah
so
like
it
they'll
be
wonderful.
I
think
we
have
a
couple
of
items
on
there
to
like
figure
out
any
remaining
blockers,
but
I
think
yeah
that
sort
of
timing
of
doing
it
next
week
would
be
great.
A
I
think
there
are
a
couple
of
things
if
we
hope
to
squeeze
in
under
the
wire,
but
if
we
don't
squeeze
them
in,
I
agree
that
it
is
probably
time
I
wanted
to
therefore
move
us
on
to
our
open
discussion
topics.
The
first
one
is
a
perhaps
scientific
one
from
peter
to
decide
on
the
next
alternative
time.
Meeting.
A
Absolutely
so
we're
not
yet
replacing
this
time
slot
and
we're
talking
about
our
out-of-band
meetings
now
that
we
have
exited
october
or
october,
are
there
any
proposals
that
we
should
that
we
are
so
choosing
between
her.
A
Yeah,
I
didn't
know
what
they
wanted
to
try
to
find
like
try
to
pick,
maybe
a
different
one
that
you
know
it
feels
like
they
all
feel
like
similar
times.
I
don't
know
if
we
want
to
try
to
pick
a
different
time
that
to
see
if
we
do
get
other
people
showing
up
as
it
were,
but
I
I'm
happy
with
all
the
times
worked
great
for
me,
which
is
not
a
prerequisite,
but
they
did
all
work
for
me.
So
I'm
I'm
amenable.
C
A
I
mean,
I
think,
the
one
the
the
the
geography
which
I
think
we
might
want
to
see
if
there
are
people
on
is
sort
of
the
asia
geography
which
I
think
we
basically
exclude
with
this
time
zone.
Maybe
that's
not
true
like
right
now
it
is,
I
don't
know
it
is
midnight
in
china,
like
east
china,
for
example,.
C
A
We
don't
have
a
solid
proposal
for
asia-friendly
times
and
we
can
try
to
see
if
they're,
if
one
no
rises
and
if
there's
any
demand,
maybe
we
can
ask
in
cups
users,
but
in
the
meantime,
why
don't
we
pick
one
that's
more
in
our
standard
menu
that
we've
chosen
from
and
I
I
am
happy
to
go
with
whatever,
whatever
time
other
people
would
like.
A
I
think
we
were
saying
yes,
no,
I
mean
we
did
9
a.m
and
10
a.m.
I
think
we
did
both
of
those
times.
Those
are
eastern,
so
6
a.m
and
7
a.m.
Pacific.
I
can't
really
do
any
other
times.
C
A
Tuesday
or
thursday
or
wednesday,
it's
always
tricky
for
me,
with
school,
but
tuesday
or
thursday.
C
A
That
down,
I
will
send
out
yeah.
I
guess
I'll,
send
that
the
same
invite
thank
you
and
we
can.
We
can
see
if
there's
anyone
that
is
in
a
it,
proposes
a
sort
of
more
different
time
slot,
let's
say
9am
office
hours,
okay,
all
right
and
then
I
had
two
items
which
were
basically
about
like
figuring
out
the
119
beta
issues.
So
I
propose
we
actually
look
at
john.
Do
you
want
to
mention
your
one
first,
so
you
are
because
it's
probably
okay.
B
Yeah,
so
this
is
one
where
the
cross
controller
bootstrap
call
doesn't
work
when
it.
The
cluster
is
configured
to
use
the
api
load
balancer
from
nodes
to
the
master,
and
so
I
put
together
a
pr
which
opens
up
or
adds
another
listener
and
opens
up
security
groups.
But
the
problem
is
if
the
api
server
load.
Balancer
is
external.
The
security
groups
don't
help
because
the
traffic
is
actually
exiting
the
cluster
and
then
coming
back
in.
B
So
we
have
basically,
I
I
think
we
have
choices.
One
is
we
open
up?
You
know.
In
that
case
we
open
up
the
cops
controller
port
externally,
which
I
don't
like
the
other
possibility.
Is
we
make
people
in
this
mode
pay
for
a
second
load
balancer
so
that
they
have
an
internal
load
balancer?
In
addition
to
an
external
one
and
the
third
one?
Is
we
use
a
separate
dns
controller
managed
domain
for
the
bootstrap
protocol,
so.
D
C
I
don't,
let's
not
go
there,
so
he
tries
many
things,
that's
not
meaning
that
those
are
recommended
for
we
should
support
all
the
use
cases.
B
A
One
of
the
one
of
the
things
I've
been
thinking
about,
I'm
not
saying
we
should
make
this
like
it's
more
of
a
directional
comment
for
the
ingress.
There
are
like
multiple
ingress
methods
and
like
potential
methods,
and
I
feel
like
we
could.
We
could
end
up
with
a
list
of
of
ingress
for
the
api
server
right.
You
could
have
a
a
dns.
You
could
have
something
that's
only
accessible
over
vpn.
You
could
have
a
load
balancer.
A
You
could
end
up
with
like
different
parameters
for
each
and
I'm
thinking
mentally
that
we
we
want
to
maybe
trend
towards
that
or
even
one
day
separating
them
out
of
the
cluster
which
might
make
it
like.
Obviously,
that's
a
much
longer
term
thing,
and
maybe
the
same
thing
applies
for
internal.
Maybe
we
at
least
mentally
have
a
list
of
these
things,
and
we
that
list
the
internal
one
is
completely
separate
from
the
external
one,
and
then
we
somehow
enable
them
to
share
a
load
balancer
for
the
economy
case.
B
Yeah
because
I
don't
think
we
can
use
the
kubernetes
service
discovery,
because
this
needs
to
be
up
before
kubernetes.
C
C
A
C
Okay,
I
was
under
the
impression
that
you
can
do
something,
but
okay,
then
we
will
have
quite
a
lot
of
new
things
in
there.
Anyway.
It's
probably
the
thing
to
do,
but
I
would
say:
give
access
to
zero,
zero,
zero,
zero
from
four
119
so
special
case,
and,
let's
try
to
add
an
issue
and
maybe
work
on
this
in
120
or
I
don't
know
if
you
have
a
better
idea.
A
There's
a
lot
of
echoes
of
the
the
clients
are
stuff
here.
Yeah
like.
A
Yeah,
I
feel
like
thank
you
for
raising
this.
I
feel
like
I
don't
have
a
I
don't.
I
personally
don't
have
an
obvious
answer.
I
feel
like
the
I'm
going
to
take
a
look
at
the
issue
I
feel
like
nlb.
The
second,
a
second
load
balancer
is
a
reasonable
answer,
but
I
agree
it
costs.
A
I
feel
like
the
dns
is
also
a
reasonable
answer.
I
really
worry
about
opening
the
port,
but
maybe
I
shouldn't
worry
about
that.
Maybe
we
should
just
make
sure
it's
actually
safe
to
open
the
port,
because
it's
a
better
policy
to
be
so.
It.
B
Be
we
have
a
new
option
which
says
how
you
load
balance
the
cops
controller
and
not
and.
D
D
Do
the
masters
themselves
bootstrap
through
the
cops
controller?
No
okay,
because
we
did
have
with
the
cert
issue
we're
having
limitations?
Where,
if
you
have
your
load,
balancer
be
internal,
you
can't
have
it
be
an
nlb
and
views
for
api
because
you
can't
tear,
pin
requests
so
traffic
from
the
masters
can't
go
through
the
nlds.
If
they're
internal.
A
Do
we
do
we
have
that
one
written
down
by
the
way?
Because
that
issue
written
down,
because
that
was
we,
I
think
that
one's
come
before
and
we
thought
it
was
fixed.
But
maybe
maybe
it
wasn't
fixed.
D
So
my
my
cert
request
includes
api
validation
for
those
scenarios.
C
And
it
will
be
very
awkward
if
people
run
cube
ctl
on
the
masters
and
well
end
up
on
the
doing
the
hairpinning
thing
and
the
api
not
responding.
C
C
So
when
you
enable
use
for
internal
api,
it
does
two
things.
One
is
set
the
dns
the
api
address
on
the
cubelet
and
one
said
the
dns
records
we
could
still
set
the
dns
records
and
anything
on
the
worker
nodes
or
whatever
in
there
would
go
to
the
load
balancer,
but
the
cubelet
could
be
made
less
load
balanced,
go
to
dns
dns,
go
to
local.
A
I'm
not
gonna
be.
Are
we
talking
about
john?
I
was
talking
about
the
issue
that
john
raised.
Are
we
are
we
talking
about
the
certificate
issue?
At
this
point,
I
think
we're
moving
to
certain
okay,
sorry,
the
well
on
john's
issue.
One
thing:
sorry,
just
one
thing
that
occurred
to
me:
we
control
the
client
when
it's
talking
to
cops
controller,
so
we
can
set
our
own
like
http,
retry
policies,
or
things
like
that.
So
that
is
that
is
one
potentially
saving
grace.
We
have
there
that
we
are
much.
A
We
are
much
more
liberty
to
do.
Fancy
stuff.
I'd
say
like
if
the,
if
the
reason
that
people
don't
like
dns
is
because
of
the
propagation
delay,
we
could
introduce
a
much
more
aggressive.
We
first
of
all,
we
could
just
bypass
dns
and
just
like
look
it
up
in
ec2,
I'm
not
very
keen
on
that,
but
we
could
also
like
introduce
much
much
faster
tcp
retries
of
the
different
hp
retries
of
the
different
ip
addresses
and
things
like.
A
A
We
certainly
want
to
split
the
we
want
to
split
the
options
we
don't
want
to.
We
would
like
to
we
like:
don't
want
to
honor
the
option,
that
is,
that
doesn't
work
and
yeah,
whether
we
need
it,
whether
we
need
a
separate
field
or
whether
we
can
find
some
strategy.
That
actually
is
fine
like
that.
A
C
C
C
B
I
I
thought
the
point
was
for
the
worker
node
cubelets,
the
master
cubelets
should
go
127.001
all
the
time.
I
would
think
because
you
know
you
got
an
api
server.
There.
A
That's
the
way
we've
built
it
anyway.
Yes,
the
there
is
a
separate
discussion,
there's
a
separate
argument
about
whether
we
should
cross,
but
that's
that's
a
much
bigger
change.
A
So
I
think
I
think,
the
I
think
on
the
master
that
as
john
says,
that
cubelets
should
already
have
1270
the
localhost
address,
regardless
of
any
setting
that
I'm
aware
of
okay,
there
are
so
many
exactly,
but
no,
I
think
I
think
that's
always
the
case
that,
like
we
don't
the
masters
are
basically
self-contained
each
each
each
control,
plane
node
is
self-contained
and
talks
only
to
itself,
and
we
don't,
for
example,
have
fcd
edc
is
the
one
where,
like.
A
A
A
Cool,
I
don't
know
if
there's
anything
more
here,
it
sounds
like
there's.
Definitely
some
investigation
to
be
done,
some
thinking
to
be
done.
I
agree,
john,
that,
like
with
your
notes
there
about
like
moving
to
a
separate
option
and
maybe
just
defaulting
but
or
splitting
the
options
should
we
move
on.
It
looks
like
I
don't
know
if
ollie's
here
only
added
something
since
the
movie.
D
C
A
Well,
this
is
very
interesting.
Can
you
can
you
describe
the
pr
just
while
we're
while
I'm
bringing
it
up?
It's.
C
Not
a
pr
it's
an
issue,
so
my
understanding
is
that
for
web
hooks
and
metrics
server
there
is
some
pro
generating
the.
B
This
is
basically
the
need
to
get
the
tls
primarily
tls
server
shirts,
but
for
components
that
have
basically
n
cluster
domains.
So
basically
you
want
a
tls
server
cert
that
is
signed.
Oh
oh,
I
think
you
want
one
that's
signed
by
the
cluster
ca,
but
you
definitely
want
one.
That's
trusted
by
things
like
the
api
server
itself,
and
I
also
have
use
cases
you
just
want
to
do.
Tls
within
the
cluster.
B
You
want
it
there,
so
I
wrote
or
no
I
I
found
and
then
kind
of
heavily
modified
a
an
approver
which
uses
the
csr
api
of
kubernetes
and
also
a
sidecar,
which
then
makes
the
request
for
this.
So
with
those
components
you
can
actually
a
with
a
service
account,
you
can
get
a
cert
for
the
pod
and
all
the
services
that
that
pod
is
a
member.
B
D
B
Yeah
I
have
this
as
sort
of
a
set
of
separate
projects
and
I
think
ola
is
trying
to
do
something.
B
A
That
what
you've
described
johnson's,
I
mean
separately
from
this
issue-
that's
really
cool.
I
I
I
want
that
the!
How
do
you,
how
do
you
know
which
services,
or
which
dns
names
effectively
a
service
account,
is
allowed
to
speak.
B
B
So
so,
basically,
the
pod
lists
all
the
services
that
I
did
in
the
namespace
and
finds
out
what
I
I
can't
even
remember
what
calls
I
do,
but
I
basically
look
at
the
services
and
see
what
maps
to
that
pod.
B
Generate
the
csr,
then
I
also
identify
the
pod
in
the
common
name,
which
is
also
another
problem
with
the
csr
request.
Is
I
can't
give
a
any
sort
of
information?
That's
not
in
the
csr
other
than
what
my
authenticated
identity
is
and
then
the
approver
then
goes
and
queries
the
api
server
to
validate
all
that.
A
Cool,
I
think
we
looked
at
this
by
the
way.
There
is
a
trick
where
you
can
add
additional
sections
to
the
csr,
which
is,
I
think,
it's
done
by
the
gcp
approver,
and
it's
like
a
little
bit
weird.
Did
we
look
at
this
together
peter?
I
can't
remember.
B
A
A
It's
like
a
different
so
like
the
csr
is
like.
I
don't
know
what
it
is.
It's
a
multi-part
mime
doc
anyway,
it's
like
it's
a
pam,
a
pam,
it's
a
pem
thing
and
you
can
have
multiple
spam
sections
and
like
one
of
them,
is
a
csr
and
then
like
there's
this
other
one
and
like
the
one
that
I've
seen
is
google
going
and
like
registering
with
whoever
the
authority
is
like
some
additional
section
anyway.
B
A
B
Can
put
information
there
but
yeah
I
did
bring
up
to
sigoff
and
I
I
think
they
understand
the
use
case,
but
it
would
be
nice
if
an
approver
could
say.
Okay,
I
want
you
regardless
of
what's
in
the
csr.
I
want
you
to
issue
this
search,
but,
yes,
that
would
simplify
things
greatly,
but.
B
The
other
problem
I
found
is
for
some
of
the
web
hooks.
The
api
server
expects
the
cert
to
have
an
unqualified
domain
in
the
cert.
E
A
A
Right-
let's
not
talk
about
that.
Let's
talk
about
that
some
other
time.
E
B
Yeah
so
so
I
had
to,
I
had
to
loosen
the
security
of
my
approver
to
be
willing
to
issue
unqualified
domains
in
the
certs,
because
otherwise
I
could
not.
A
I
think
your,
I
think
your
your
projects,
sound
wonderful,
I'm
definitely
going
to
take
a
look
at
them.
Yeah.
Can
we
circle
back
to
olay's?
I
think
then
ollie
was
saying
that
this
was,
I
guess,
a
heavy
dependency
to
take,
which
I
mean
it
sounds
like
a
good
dependency.
I
don't
think
that's
unfair
to
say
that
it
would
be
a
it
would
be
a
dependency,
and
then
you
want
some
sort
of.
B
Yeah
he
wants,
he
wants
to
do
a
crd,
and
so
you
do
a
cd
and
it
it
looks
like
it
would
then
maintain
a
signed
cert
and
a
secret
somewhere.
A
Yes,
but
presumably,
if
I
can
read
secrets,
I
can
steal
the
service
account
key
and
okay
anyway,
right,
okay,
but
I
mean
I
think.
A
B
No,
it's
not
a
certain
manager
thing,
cert
manager,
you
have
a
crd
which
specifies
a
certain
in
it.
Oh
I
swear
you
use
the
certain
secret
and
what
I
do
is
you
specify
you
want
and
you
get
it
in
the
volume
in
your
pod.
A
Perhaps
that
so
perhaps
for
olay
we
could
say:
have
you
considered
cert
manager.
A
B
Can't
get
certs
from
the
kubernetes
ca,
but
it.
A
And
it
doesn't
solve.
I
really
like
your
model,
though,
of
the
the
volume
where
it
doesn't
get
persisted
into
a
secret.
That's
nice.
D
A
Volume,
oh
and
I
think
actually
what
ollie
was
saying
was
we
just
didn't?
We
don't
need
the
admission
controller
here
for
for
his
particular
use
case,
which
I
think
is
good
right
so,
like
I
think,
he's
just
saying
that
csrs
are
more
complex
than
maybe
they
need
to
be,
and
it
doesn't
even
sound
like
you.
It
sounds
like
you've
also
hit
some
problems
with
csrs
or
some
challenges
with
csrs.
B
Yeah
yeah
because
yeah,
because
the
csr
api
is
missing,
features.
A
A
B
A
A
But
the
I
mean,
I
think
also
like
you
know,
we
don't
require
synchronous
communication
in
general
of
our
computer,
so
we
can
hopefully
have
a
meaningful
conversation
on
this
and
then
like,
but
yeah
face-to-face
is
often
more
efficient,
okay.
So
the
the
only
remaining
two
things
on
the
agenda
that
I
see
are
the
two
that
I
added,
which
are
about.
B
Actually,
I
have
one
more
thing
to
point
out
is:
maybe
somebody
from
sigoth
might
be
interested
in
joining
the
conversation,
but.
A
That
is
a
good
point.
Yeah.
A
Yeah
I
mean
it'd
be
great.
If,
like
the
csr
api
was
we
decided
whether
to
make
the
csr
app
a
little
bit
more
flexible
or
like
we
get
a
verdict
from
them
that
they
are
not
going
to
extend
the
csr
api
in
any
way,
shape
or
form,
and
so
that
you
know
we.
We
are
right
to
build
something
else,
and
then
it's
a
question
of
where
it
belongs
exactly
loop
in
sick
off,
okay,
so
the
unless
there's
anything
else,
anyone
wants
to
bring
up
the
other
items
are
around
119
zero
beta
one.
A
Okay,
so
I
I
mean
I
I
put
two
items
on
here,
but
I
think
other
than
client
cert.
There
are
a
couple
of
things
we
want
to
squeeze
in,
but
I
I
believe
we're
looking
good
for
the
for
the
to
do
a
beta
sometime
next
week,
ish
early
next
week,
hopefully
and
probably
have
to
carve
out
it,
looks
like
we
will
probably
have
to
carve
out
some
exclusions
for
some
things,
and
then
I
thought
we
could
like
the
most
notable
one
is
client
search
strategy.
A
A
I
think
I
think
that
most
of
them
are
nice
to
haves,
except
for
the
first
four
I
guess
in
the
existing
blockers
list,
and
I
think
I
want
to
do
client
search
separately,
but
the
other,
like
I
feel
like
we
like
rodrigo,
oh
rodrigo,
was
looking
here
he's
looking
after
this
first
one
and
he
writes
a
comment
that
he
he
had
a
bunch
of
merge
conflicts,
cherry
picking
to
118,
but
he's
going
to
try
to
address
them.
A
So
I'm
assuming
that
that
means
that
the
pr
itself
merged,
which
I'm
just
checking
yes
and
cherry,
picked
at
119.,
perfect
and
so
he's
just
referring
to
tripping
to
118..
Okay.
So
that's
wonderful!
So
that's
not!
Even
so.
We
can
almost
cross
that
off
the
list
in
terms
of
119
block,
word
block
blocking
I'm
going
to
look
at
this
issue
about
clusters.
A
Sorry
I
didn't
know,
there's
someone
nope!
Okay,
I
don't
I'm
always
worried,
I'm
talking
over
people.
So
if
I
am
just
shout
louder
the
cluster,
I
cannot
start
with
older
version
of
kubernetes
I'm
I
was
going
to
look
at
this.
I
did
not
have
the
chance.
I
will
take
a
look
at
this
still.
I
believe
the
answer
is
that
I
just
should
do
some
new
images,
some
of
the
amis
and
I'm
gonna
do
so.
A
The
clients
are
one
I'd
like
to
deal
with
last
and
the
max
price
and
mixed
instance
policy
errors
with
launch
templates.
I
think
it's
still
open.
I
don't
know
if
anyone's
looking
at
this,
if
no
one
else
is.
D
There's
a
duplicate
issue
and
I
left
a
few
comments
in
the
other
one.
It
seems
like
like
what
their
instance
group
spec
looks
defined
correctly,
such
that
it
shouldn't
run
into
this
error,
which
makes
me
wonder
if
something's
happening
like
the
auto
scaling
group
is
using
the
wrong
version
of
the
launch
template
or
maybe,
if,
like
nodes,
are
like
the
auto
scaling
group
needs
to
be
rolled
before
something
can
be
updated
like
that.
I'm
not
sure.
C
I
can
look
into
that
after
merging
the
launch
template
versions,
because
I'm
still
touching
that
code,
so
I
would
know
where
to
look,
but
probably
I
will
try
to
reproduce
it
also
see
if
I
can
get
into
it.
I
think
I
noticed
some
similar
things
from
time
to
time,
but
couldn't
reproduce
easily
and
was
hunting
something
else
at
the
time.
So
if
they
come
up
with
some
way
of
reproducing
the
issue,
then
I
guess
it's
easier
to
fix.
A
I'll
also
try
to
reproduce
it.
I
think
that'll
be
good
and
I
think
I'm
not
currently
using
spot
instances.
So
I
would
like
to
keep
using
smart
instances
because
it's
cheaper,
so
I
mean
whatever
they're
now
called
max
price.
I
guess
the
yeah,
but
thank
you
sabrina.
That's
wonderful
looks
like
the
next
one
I'll
just
traverse
the
others
like
the
next
one
is
nlb
support
which
looks
like
it
has
merged.
So
thank
you
to
all
of
your
work
on
that
and
it
was
potentially
blocking
clients.
A
So
we'll
come
back
to
that
in
a
minute.
There
are
still
the
well
there's
three
three
cd
four
ltd
manager,
pr's,
which
I
will
take
a
look
at.
I
think
we've
unblocked
the
arm
one,
hopefully
cyprian.
If
you
want
to
have
a
look
at
the
approach,
I
took
four.
C
A
Yes,
we
can,
I
think
we
can
do
that.
I
meant
blocked
as
in
like,
as
I
understand
it,
the
current
arm64
pr
puts
us
on
an
image
which
isn't
entirely
supported,
and
so
we
have
a
strategy
for
getting
off
of
that
now.
A
C
Okay,
thank
you,
yeah.
We'll
look
at
that.
I
think
I
already
looked
at
something
similar
some
months
ago
when
I
was
trying
to
build
a
cd
manager
by
the
way,
the
lcd
to
something
makes
it
really
annoying
to
build.
A
C
You
are
using
a
special
lcd
version
patched
because
the
two
to
something
when
they
released
it
was
broken
and
they
never
fixed
it
or
something.
A
A
Old,
it
is
one
like
no
one.
No
one
should
be
using
it
today.
If
they
can
afford,
they
have
any
way
to
avoid
it.
I
will,
I
will
have
a
go
on
the
mac,
it
it.
It
should
be
reproducible.
It
may
not
be
reproducible
on
all
hardware.
It
sounds
like
so
it
might
be
that,
like
the
workaround
might
be
to
build
it
inside
of
docker
on
your
mac
and
make
sure
you're
plugged
into
the
wall
before
you
do
that,
but.
C
A
And
go
ahead!
Sorry.
A
Okay,
the
it
looks
like
three
pr's
have
also
merged
and
then
there's
another
one
that
has
been
raised
as
a
blocker.
There
was
no
name
on
it.
I
didn't
see
who
typed
it
I
did,
which
is
about
terraform
migration
from
launch
configuration
to
launch
templates
right.
D
So
the
two
options
that
I
came
up
with
are
either
have
terraform,
define
both
or
sorry
have
cops,
define
both
launch
configuration
and
launch
template,
or
we
have
something
in
the
release.
Notes
that
says:
remove
the
launch
configuration
from
the
terraform
state
manually
before
doing
the
terraform
apply.
C
C
C
D
Well,
yeah,
I
think
I
misspoke
it's
reading
the
issue
more.
It's
not
that
the
nodes
need
to
be
rolled.
It's
that
the
auto
scaling
group
terraform
tries
to
delete
the
launch
configuration
before
the
auto
scaling
group
has
been
updated
to
use
the
launch
template
and
there's
no
way
that
we
can
define
that
kind
of
dependency
tree
in
terraform
to
ensure
that
that
happens
in
the
correct
order.
D
Well,
the
apply
fails
the
command
errors
out.
I
don't
know
if
it
eventually
succeeds,
if
you
retry
it
or
not,.
A
I
think
it's
definitely
a
something
we
should
treat
as
a
like
something
we
want
to
figure
out
the
beta.
So
I
think
that's
thank
you
for
putting
that
in
there.
Like
that's
good,
I
think
I
actually
thought
that
what
you
said
was
interesting,
like
I
thought:
terraform
had
fairly
good
like
sequencing
overrides,
so
maybe
there
is
something
I
just
don't
know
if
we
can
mess
around
that,
but
there's
also
a
script
in
there.
So
I
think
this
is.
A
This
is
a
good
issue
and
we
can
look
at
that
and
treat
it
as
a
likely
blocker
or
worst
case.
We
have
a
workaround
from
looks
like
users,
the
the
one
that
I
definitely
wanted
to
make
time
for
was
understanding
our
strategy
for
client
certificate
authentication.
I
tried
explaining
this
to
classical
lifecycle
and
then
my
sort
of
update
to
that
thing
and
sort
of
got
tied
myself
in
nuts.
So
I
don't
know
if
our
current
strategy
is.
Can
we
would
you
mind
reminding
me
peter.
D
Yes,
so
the
issue
is
that
if,
if
a
user
has
specified
an
acm
certificate
for
the
api
load,
balancer
that
effectively
breaks
the
client
certificate,
authentication
that
we
were
thinking,
we
were
relying
on
for
the
cops
export
cube
config.
D
The
solution
that
we
came
up
with
was
requiring
that
anyone
specifying
an
acm,
cert
and
upgrading
to
119
needs
to
use
an
nlb
instead
of
a
classic
load
balancer
and
then
what
we
can
do
is
create
a
second
listener,
a
second
port
on
the
load
balancer.
D
That
does
not
have
the
acm
cert
on
it
and
it's
instead
doing
a
tcp
layer,
4
pass-through
back
to
the
api
server,
so
that
the
tls
session
is
established
correctly
with
the
client
certificate,
and
so
the
pull
request
that
I
linked
to
effectively
implements
that.
I
was
having
issues
testing
it
in
prow.
D
I
had
a.
I
had
proud
requesting
an
acm,
cert
and
provisioning
an
nlb
but
prowl,
for
whatever
reason
couldn't
like
cops
validate
cluster
could
connect
to
the
api
server
correctly.
But
then
the
cube
control
commands
would
fail,
saying
connection
refused,
which
seems
like
a
very
strange
error
for
a
network
load
balancer.
D
It
wouldn't
be
security
group
related
or
that
would
be
a
timeout,
and
I
ensured
that
the
ports
were
listening.
So
I'm
not
sure
how
connection
reviews
happened,
but
cyprian
was
able
to
test
it
locally
and
it
seemed
to
work.
So
the
the
pull
request
implements
kind
of
the
basic
support
for
that.
It's
not
like.
Not
all
of
the
reconcile
state
transitions
are
implemented
correctly
like.
D
If
you
delete
certain
things
out
of
band,
they
may
not
get
fixed
correctly,
but
at
least,
if
you're
only
touching
things
through
the
cluster
spec,
it
should
work
correctly.
D
D
A
Thank
you
very
much.
I
need
you
to
untie
me
I'll
my
nuts
again,
but
that
is
great.
Thank
you.
The
on
the
export
cube
conflict.
I
would
treat
that
as
a
as
a
sort
of
atomic
operation,
so
I
wouldn't
expect
users
to
export
coupe
config
and
then
like
manually
change,
the
host,
for
example.
So
that's
how
I
think
about
it.
So
if
you're
gonna,
if
you're
gonna
set
a
dns
name,
that
is
the
acm
dns
name,
then
there's
no
point
in
exporting
the
client
config
in
there
for
you
the
client's
certificate.
A
Therefore
you
should
not
the
everything
else
sounds
great.
I
think
the
the
I
guess
we
should
ask
users
whether
they
like
something
camp
was
just
like.
Why
do
users
use
acm
and
I
tried
to
say
it
was
because-
or
I
offered
the
reason
it
was
because
then
they
don't
have
to
worry
about
certificates
and
like
holding
the
private
key
themselves,
which
I
think
is
true.
D
A
This
it's
a
real
dns
name
as
well.
Yes,
although
now
we're
gonna
have
to
do
that
again
effectively.
Oh,
unless
you
use
some
user
authentication
system.
Okay-
and
I
guess
the
other
questions
are
usually
gonna
be-
is
this
gonna
be
a
default
thing
or
are
you
just
gonna
have
to
like
set
a
field
to
to
turn
on
the
second
port.
D
A
I
think
that's
great
for
the
beta.
I
can
see
users
saying
we
don't
want
to
like
open
this
at
all,
because
we
always
want
to
use
this,
but
I
think
that's.
It
sounds
like
that.
Pr
will
will
get
us
to
beta
and
that
will
be
perfect.
Then
we
can
get
some
feedback
on
whether
users
are
happy
to
open
that
that
port
or
not.
D
A
A
We
could
create
a
static,
long
live
forever
token
that,
like
an
enabled
token
off,
I
feel
like
I
feel
like
we
fought
so
hard
to
get
rid
of
token
off,
I
think,
or
at
least
mostly
get
rid
of
it
and
and
if
basic
oauth
is
going
to
help
won't
be
too
far
behind
behind.
So
I
like
I
like
the
approach
that
we
are
pursuing,
but
I
wanted
to
make
sure
we
were
covering
it
all.
D
Yeah
and
then
regarding
the
nlb
support
itself,
it's
supported.
D
The
migration
works
with
downtime
of
a
few
minutes
because
of
the
we
didn't
support,
the
aha,
where
you
can
specify
creating
both
load
bouncers
and
then
it
doesn't
clean
up
the
old
load
balancer.
A
The
risked
as
in
like
the
biggest
problems
are
done.
Thank
you
to
peter
and
everyone
that
worked
on
those,
and
so
we
should
be
able
to
do
a
beta,
ideally
next
week
with
some
and
there
will
be
some
release
notes
and
I
think
that's,
okay.
If
people
agree.
D
A
C
Sorry
quick
question:
I'm
I
tried
to
add
the
ebs
kms
key
for
encrypting,
dbs
and
bumped
into
a
small
issue
where,
when
we
compare
expected
and
actual
values,
cops
ignores
expected
values
which
are
null.
So
if
your
setting
or
whatever
is.
C
C
A
Yeah,
so
like
the
the
difference,
if
it
was
on
a
new
cluster,
the
difference
would
be
null
means.
Do
whatever
is
the
right
thing
in
terms
of
our
like
latest
recommendation,
empty
string
would
mean
I
want
the
empty
string
and
a
value
would
mean
I
want
that
value
like
the
so,
in
other
words,
it's
the
difference
between
like
a
boolean
is
easier
right
like
true
means,
yes,
false
means
no
and
null
means
true
or
false,
based
on
what
you
recommend
like
at
the
time.
A
So
that's
sort
of
why
that's
why
we
draw
the
distinction
between
null
and
empty?
Sometimes,
this
is
obviously
not
great
ux,
particularly
if
you've
already
created
or
bound
it
down
the
key.
I
suggest
we
as
we're
over
time.
Let's
talk
about
it
next
time
and
and
maybe
follow
in
file
on
thursday
and
follow
an
issue,
and
if
we
want
to
do
it
before,
then,
how
about
that.
D
A
All
right,
any
last
items
otherwise
see
you
all
in
two
weeks
or
on
thursday
at
9
00
a.m.
Eastern
6
a.m,
pacific,
and
I
will
put
that
on
calendars
and
have
a
wonderful
weekend.