►
Description
Meeting Notes: https://docs.google.com/document/d/16CEsBSSGm3sMpvB_cFnKnqqi1OxhIcyX3lVwBpIyMHc/edit#heading=h.65cg9rbfsql9
Highlights:
- HA
- Cert rotation
- Self hosting
A
Hello
today
is
Wednesday
January
10th,
and
this
is
the
Syd
cluster
lifecycle,
breakout
meeting
or
office
hours
we're
in
round
Colin
regarding
high
availability,
self-hosting
and
upgrades
I
added
a
couple
of
things
to
the
agenda
item
to
talk
about
today
and
we
might
sprawl
out
from
there.
One
of
the
topics
that
we
discussed
in
the
last
cluster
life
cycle
meeting
was
the
notion
that,
in
order
for
us
to
get
to
GA,
we
don't
necessarily
have
to
have
high
availability.
A
You
thought
out
as
a
turnkey
solution,
so
long
as
the
documentation
is
all
in
place
and
we
have
a
start
on
some
of
the
documentation
that
exists
there,
but
I'm
looking
for
takers
to
follow
up
on
potentially
addressing
some
of
the
other
node
constraints,
especially
regarding
upgrades
right.
So
that
is
a
piece
that
has
been
missing
from
the
documentation.
A
Yes,
it's
specifically
geared
around
Covidien
higher
availability
stuff
for
leveraging
existing
Covidien
installs
for
H
a
and
having
the
upgrade
instructions
be
present
there.
We
have
the
installation
instructions
for
how
to
use
it,
and
we
will
eventually
have
a
turn,
a
much
more
turnkey
solution,
and
some
of
this
will
go
into
in
the
next
bit
when
we
talk
about
some
of
the
work
that
fabrini
has
been
doing.
But
for
the
time
being,
in
order
for
us
to
get
to
GA,
we
want
at
least
have
everything
documented.
C
In
that
Google
Docs
document
that
I
created
about
H,
a
and
Q
ad
and
there's
eventually
going
to
be
some
some
information
about
upgrades,
because
that's
definitely
something
that
I
have
to
look
into
for
my
work
and
I'm
going
to
document
that
there.
But
obviously
somebody
has
to
take
it
over
to
the
well
official
kind
of
documentation
that
Jaime
started.
Doing
yes,.
B
A
A
Have
it
as
one
of
the
following
items
and
kind
of
repeat
it
over
and
over
again,
and
as
people
eventually
have
cycles
being
able
to
take
a
look
at
it
and
contribute
is,
is
useful
so
which
will
probably
find
me
doing
over
the
course
of
110
and
111
is
repeating
myself
over
and
over
again
again
regarding
some
PRS
and
issues
so
that
that's
probably
the
first
item.
The
second
thing
I
want
to
talk
about
with
regards
to
H
a
is
some
of
the
work
that
Fabricio
has
done.
A
I
don't
know
if
people
have
had
a
chance
to
take
a
look
at
it
at
all,
but
it's
good
stuff
for
me
too,
and
I
know
he's
on
the
call,
if
you
can
want
to
provide
some
links
to
the
stuff
that
you
had
sent
a
while
ago.
So
other
folks,
now
that
everybody's
back
in
the
New
Year,
so
the
folks
can
take
a
look
at
it.
A
An
email
response
to
your
original
prototype
and
I
don't
see
any
reason
why
we
can't
start
to
feature
gate
some
prototypes
for
people
to
tinker
around
with
I,
don't
know
if
people
have
thoughts
or
feelings
about
that.
But
I
thought
is
the
workflow
that
you
created
for
your
prototype
was
actually
kind
of
nice
and
clean
so
and
allow
for
other
people
to
tinker
with
it
want
to
talk
to
the
four
beats
you.
D
D
Working
pretty
well
and
then
I
will
transform
my
prototype
into
a
PR
where
the
people
can
look
at
it,
and
second
I
will
put
into
the
PR
all
the
evidence
limitation
that
I
find
out
working
on
on
this
prototype,
and
we
will
see
if
those
limitations
are
acceptable
or
not
to
make
this
PR
becomes
become
something
under
feature.
Gate.
A
Arts
in
the
know,
about
Febreze
his
prototype
we've
been
discussing
for
a
long
time
about
how
we
expect
the
Canadian
workflow
to
kind
of
work
for
joining
masters,
and
this
is
a
prototype
with
regards
to
how
you
would
join
masters
to
an
existing
cluster
and
there's.
There
are
some
gotchas
and
details
there
like.
A
How
do
you
deal
with
secrets
and
everything
else
and
that
that
is
part
of
what
we'll
have
to
address
with
with
the
PR,
so
I'm
sure
there
will
be
comments
and
details
for
it's
coming
pretty
soon,
but
the
workflow
when
I,
walked
your
workflow
and
took
a
look
at
it.
Looked
nice
I
liked
it
a
lot.
I
did
have
some
thoughts
on
it,
but
I
totally
don't
remember
what
they
were
now
because
there's
a
while
ago,
but
the
idea
is
basically
just
have
like
a
very
simple
command
like
join
and
then
having
a
master.
A
A
D
And
I,
availability
and
Intel
in
that
PR
Jamie
is
using
as
advertised
address
for
both
the
Masters,
the
load,
balancer
EP,
yes
and
I.
I
was
wondering
if
this
is
the
right
solution,
because
if
we
do
this,
for
instance,
we
will
have
that
both
the
controller
manager
and
all
the
thought
that
goes
through
the
kubernetes
service
will
go
out
from
the
cluster
pass
through
the
load,
balancer
and
then
eat
on
a
bi
server.
Yes,.
D
A
There's
a
couple
ways:
we
can
do
this,
the
the
default
configuration
I
mean
this
is
the
problem
where
high
availability
becomes
fractal
and
we
have
at
some
point
in
time.
We
need
to
just
like
document
one
way
of
doing
this.
The
the
way
that
Jamie
had
documented
is
the
well
tread
and
trodden
path
between
multiple
implementations
and
they're.
A
Ideally,
in
the
future,
we
have
discussed
the
idea
of
potentially
having
something
like
envoy
as
a
sidecar
to
all
of
the
components
in
the
system,
so
that
I
could
talk
to
local
host
and
then
just
resolve
the
load.
Balancing
of
the
master
is
Tehran
boy
like
and
there's
multiple
ways
that
you
could
do
this.
A
A
Is
one
of
the
simpler
configurations
and
that's
the
reason
why
we
documented
it
but
I,
don't
know.
There's
there's
been
this
long-standing
model
that
has
existed
in
Ilia.
You
can
chime
in
here
too,
that
came
originally
from
Luke
that
the
the
edict
was
that
eventually,
ideally,
we
would
be
able
to
provide
a
turnkey
solution
from
Kubb
ATM
that
didn't
have
dependency
on
external
facilities,
such
as
cloud
provider,
load,
balancers,.
E
F
E
E
B
A
D
G
I
didn't
know
that
this
was
supported
on
upgrade,
but
basically
like
Lucas
describes
kind
of
in
this
issue
that
the
mechanism
is
supposed
to
on
upgrade
if
the
certs
are
old,
enough,
move
them
out
into
a
backup
directory
and
then
run
cooler,
idiom
face
certs
to
regenerate
things,
and
it's
done
as
a
post
upgrade
task,
and
it
seems
like
this
was
a
feature
that
kind
of
is
just
supposed
to
patch.
The
fact
that
we
didn't
think
about
certs
rotation
linked
in
this
issue
is.
A
G
G
A
G
Yeah
I
think
I'm
like
pretty
much
of
the
certs.
Aren't
there
then
and
the
CA
is
there,
then
the
certs
are
going
to
get
generated,
so
you
can
kind
of
just
move
them
just
like
what
Lucas
is
describing
and
I
haven't
like
read
thoroughly
through
Peters
modifications
that
this
stuff
is
merged
and
I'm,
just
like
wondering
if
this
is
something
that
we
should
expect
like
when
were
to
to
be
implementing.
G
You
know,
like
I,
know:
buddy
really
mentioned
it
until
Peter
was
like
by
the
way
I
did
this
thing
for
the
API
server
to
support,
auto
rotation
of
certs
on
upgrade
and
it's
merged,
and
is
this
something
that
like
is
this?
The
way
that
we
want
to
do?
This
should
I
put
in
a
PR
like
an
additional
PR
for
the
Etsy
DTLS
stuff?
How
do
we
want
this
to
work?
Yes,.
A
A
So
once
you
get
to
self
hosting,
you
have
the
magic
of
being
able
to
just
update
your
secrets
on
the
fly
right,
but
that
that
comes
with
all
kinds
of
security
constraints
and
I
know
that
there's
been
issues
along
the
way
with
sig
off
I.
Think
it's
totally
reasonable
for
the
time
being
and
people
can
talk
against
it.
A
G
Just
talking
with
the
Peter
yesterday,
he
was
saying
like
this
was
definitely
it's.
It's
not
like
part
of
the
MVP
that
we
need
for
Etsy
DTLS,
but
I.
Think
that,
like
if
we
don't
implement
this
now,
then
we're
gonna
be
in
the
same
boat.
You
know
a
year
from
now
like
where
none
of
these
thought
about
it
or
volunteered
to
work
on
it
and
like
now
our
certs
are
gonna
expire
yeah.
A
G
A
G
You're
talking
about
the
extra
sands,
yes
right,
you
can,
you
could
pass
the
with
the
sed
extra
orgs.
All
right,
you
could
override
the
listen
address
from
local
hosts
to
like
eat
zero
or
something
public,
and
now
you
would
have
a
publicly
addressable
@cd
interface,
but
then
the
certs
not
going
to
have
a
SAN,
so
that
gives
the
user
a
chance
to
supply
like
the
node
host
name
or
potentially
like
a
proxy
load.
Balancer.
D
G
Yeah
and
then
also
I
was
actually
kind
of
see,
as
well
with
the
manual
HEA
instructions
for
using
cube
ATM
with
static
pods.
Currently
we
have
a
restriction
where
you
are
not
able
to
use
the
you
know
automatically
managed
sed,
but
if
you
were
to
pass
the
right
flags
in,
you
could
even
add
a
discovery.
Url
and
Kuby
a
DM
could
potentially
stand
up
static,
pods
on
on
multiple
masters.
G
A
G
A
A
A
There
already
exists
the
checkpoint
facilities
inside
of
the
coolant.
We
did
that
over
the
course
of
a
couple
of
releases
and
that
will
checkpoint
a
pod
to
local
disk
and
it's
a
it's
a
optional
parameter
to
the
couplet.
But
we
need
to
be
able
to
promote
this
from
an
alpha
great
feature
to
something
that's
beta
supported
as
well
as
add
the
check
pointing
capability
for
secrets.
A
The
secrets
are
temporarily,
are
encrypted
and
written
to
disk
for
being
able
to
bring
back
a
control
plane
underneath
to
failure,
Christine
conditions.
One
is
a
single
master,
self-hosted
cluster,
where,
if
that
goes
down
without
having
the
secrets,
it
doesn't
know
what
to
do
so,
it
would
checkpoint
those
secrets
to
disk
and
then
it
would
be
able
to
restart
the
control
plane
on
a
restart.
The
second
is
a
whole
data
center
failure,
condition
where
you
have
multiple
nodes
and
everything
just
goes.
The
crap.
E
Yeah
because
they
would
be
expected
to
start
on
the
same
nodes,
so
they
sort
of
they
they
become.
They
then
become
sort
of
pseudo.
Api
objects,
they're,
not
exactly
API
outfits
certain
extent,
because
in
a
way
now,
certain
truths
is
a
social
through
system
is
on
desk
list.
In
some
cases,
isn't
it
kind
of
weird.
G
A
G
A
Decided
that
it
only
needs
to
occur
on
just
the
API
server
as
long
as
the
API
server
can
come
back,
everything
else
can
come
back
afterwards,
so
you
only
need
to
add
the
weight.
Checkpointing
works.
Is
you
add
an
annotation
to
the
API
server
demon
side
pod?
That
annotation
then
flows
to
the
Kubla.
The
kulit
then
decides
that,
because
it
has
the
annotation,
it
writes
that
pod
to
disk
a
similar
thing
would
occur
for
api
server
secrets
right,
which
would
eventually
be
serving
surance
right.
Yeah.
G
A
So
part
of
the
part
of
the
lifecycle
of
a
pod
on
the
node
side
is
to
mount
any
volumes
for
that
pod,
and
that
also
includes
secrets
that
are
referenced.
Secrets
are
basically
volume
amounts
into
a
pod
itself,
so
you
can
have
explicit
mod,
so
you
can
have
config
Maps.
You
can
have
secrets
all
you,
NOPD's
PVC
claims,
secrets
and
config
maps
are
all
a
form
of
mounting
into
a
pod,
but.
A
No
because
we're
explicitly
not
adding
any
volume
outs
for
the
API
server,
so
we're
being
very
judicious
about
what
we
add
or
you
could
have
a
host
volume
out
which
allows
you
to
basically
say
I.
Have
this
well-defined
path
on
this
host
I
don't
need
to
checkpoint?
It
just
gets
mounted
from
the
host
directly
into
the
pod,
and
that
is
the
escape
hatch
that
we
have
been
debating
upon
wanting
to
do
is
do
the
host
volume
mount
for
self
hosting
yeah.
G
If
it
almost
I
mean
it'd
almost
be
like
a
like
just
a
backup
volume
like
you
could
still
use,
the
secret
still
rotate
the
secret
and
then
it
would
be
synced
with
the
host
volume
mount.
But
that
definitely
doesn't
sound
I,
don't
know,
I,
don't
know.
If
it's,
if
it's
cleaner
or
not,
then
the
secret
checkpointing
I
mean
I,
guess
it'll,
be
in
plain
text:
yeah.
G
A
Yes,
but
we've
explicitly
in
order
to
make
checkpoint
II
as
an
overloaded
word
right
and
depending
upon
your
legacy
in
history
in
grid
based
systems,
it
has
multiple
connotations
and
meanings
and
in
order
for
us
to
scope
it
down
so
that
way,
people
don't
flip
their
lid.
In
talking
about
checkpointing,
we
have
explicitly
called
it
bootstrap
checkpointing,
so
that
is
only
for
the
control
plane.
A
Now
there
are
also,
ultimately,
there
are
the
notions
of
long-running
jobs
that
want
to
be
able
to
write
State
to
a
local
disc
and
be
able
to
retro
actively
start
up
from
that
point
in
time.
Using
technologies
like
DM,
TCP
or
other
things,
and
that's
a
generic
checkpointing
thing
is
that
you
might
have
been
further
down
the
state
path
and
not
necessarily
written
your
things
to
a
volume
out
and
want
to
be
able
to
checkpoint
it
and
then
recover
from
where
you
were
last
word.
A
G
G
A
Cout
blitz
certs
to
be
able
to
encrypt
now
it's
debatable
like
I
mentioned
before
whether
or
not
we
want
to
use
host
volume
amounts,
but
then
there's
this
other
question
of
what
do
we
wanna
do
but
rotation,
because
if,
if
the
API
server
certs
are
stored
as
secrets,
it
makes
certain
rotation
way
easier.
Mm-Hmm!
That's
what.
G
A
There's
a
multiple
things
here
that
we
could
handle
like
we
could
have,
we
could
have
a
daemon
set.
You
know
that
periodically
does
the
rotation
on
disk
right
and
does
the
same
host
value
amount.
There's
there's
a
number
of
ways
we
we
can
do
this,
but
we
need
to
do
is
enumerate
the
list
of
possibilities
and
just
pick
one
at
this
point.
I
personally
am
a
little
skeptical
of
adding
secrets
checkpointing,
but
we
need
to
have
a
solution
for
self
hosting
that
allows
for
certain
rotation
than
that
host
value
mount
stuff.
A
So
it's
base
finishing
off
this
last
piece
of
the
puzzle
and
Lucas
already
has
some
of
the
pieces
in
place
for
using
secrets
that
stuff
has
been
added
a
while
ago.
So
if
there
is
a
volunteer
who
wants
to
help
work
on
this,
I
can
get
all
the
pieces
aligned
and
what
I'll
probably
do
is
open
up
an
issue
again
with
regards
to
this,
get
all
the
history
together
in
a
single
location
and
look
for
folks
who
want
to
help
drive
home
this
last
vent.
E
A
A
complicated
problem
that
has
a
lot
of
history,
like
everything
that
we're
dealing
with
now
is
a
is
one
of
those
things
where
there's
many
ways
--is
to
slice
it
and
we
have
history
behind
it.
So
what
I
think
needs
to
happen
now
is
for
me
to
try
to
help
write
down
some
of
that
history
and
some
of
those
issues
and
legacy
behind
it.
So
that
way,
other
folks
can
jump
in
and
execute
against
it
yeah.
E
A
E
Do
understand
the
whole
motivation
was
with
self
hosting,
but
I
also
understand
some
of
the
reasons
for
not
doing
it
like,
for
example,
you
know,
essentially,
if
we
do
make
it
we're
in
this
really
world
running,
to
bring
this
that's
kind
of
a
it's.
An
odd
thing
to
do.
It
doesn't
seem
like
I
mean
to
me
to
me:
does
I
mean
where's?
One
hat
on
I
could
look
at
it
that
way,
I
think.
Well,
we
we've
made
selfish
things
work
very
well,
but
now
we
just
optimize
the
system
to
run
itself
well.
A
The
biggest
motivation
there,
the
biggest
the
the
primary
driver
behind
self
hosting,
is
seamless,
upgrades
right
right,
because
then
you
could
use
upgrade
strategy,
rolling,
updates
to
basically
read
your
entire
control
plane
and
because
they're
running
on
couplets
you
can
asynchronously
update
the
couplet
and
then
modifying
your
control.
Plane
just
becomes
a
rolling
update.
That's
the
primary
driver
is
the
use
case
of
UX
of
upgrades
and
making
them
simple.
Oh
yeah,
yeah,
no,
no
you're.
E
Not
absolutely
understand
that
it's
just
sort
of
was
different.
Hassan
I
can
think
of
this
problem
differently,
but,
like
I,
mean
part
of
meals
thinks
what
about
if
we
had
a
way
of
telling
cubelet
what
to
do
other
than
just
running
static,
pods
that
I
that
are
given
his
files
right.
So
you
could
sort
of
had
some
kind
of
API
in
cubelet
that
there
would
allow
us
to
do
this,
and
then
we
could
implement
something.
On
top
of
that,
that
would
allow
us
to
do
that
through
through
the
API
server,
and
all
that
that.
A
A
A
You
can
you
can
use
the
couplet
as
a
as
a
mechanism
for
basically
executing
arbitrary
pods
in
your
environment,
provided
that
their
scope
down
right
that
meaning
of
scope
down
is
the
cool.
It
will
inherently
pick
apart
a
pod
and
try
to
mount
in
other
things,
which
requires
an
API
sir,
so
it
has
to
be
a
very
special
case.
A
I
think
what
what
the
primary
driver
is
to
give
some
history
to
is
that
boot
cube,
does
a
lot
of
this
capability
and
what
we're
kind
of
doing
here
is
distilling
down
and
making
them
first-class
citizens
within
the
core.
Some
of
the
stuff
that
boot
cube
does.
Is
it
does
its
own
separate
checkpointing,
which
is
not
actually
built
in
as
part
of
the
coolant
recovery,
startup
stuff?
G
A
Because
the
the
problem
is
semi
fractal,
there
are
many
ways
to
slice
it
right.
No,
that's
the
reason
why
this
whole
separate
meeting
exists
right
most
three
things:
each
high
availability
of
periods,
man
so
hosting
have
many
ways
to
slice
it
and
the
one
last
bit
we
need
to
manage
with
recovery
of
self
hosting,
is
managing
of
the
secrets
and
having
an
opinion
on.
E
Yeah
well
I,
just
sort
of
you
know
it's
a
I
mean
it
would
the
way
I.
Think
of
it
is
like
it's
always
not
a
bad
idea
to
sort
of
question
the
decisions
that
have
been
made
in
the
past
and
reconsider
before
we've
gone
and
spent
like
a
year
trying
to
make
this
work
nice
and
secured
yeah,
but,
like
you
know,
think
about
it,
a
bit
more
I
haven't
thought
of
it
recently
for
quite
a
while.
E
A
A
I,
don't
have
any
other
topics
to
discuss
what
I
have
action
items
that
I
want
to
follow
up
on,
one
of
which
is
to
write
down
the
history
of
some
of
the
bits
around
self-hosting
and
try
to
distill
them
down
and
I
have
an
action
item
for
next
week
to
go
through
the
backlog
of
Kubb
ATM,
so
they
could
have
mash
up
right.
There.
G
G
So
that's
good.
The
second
question
I
have
on
that
is
when
you
guys
run
across
typos,
some,
some
of
which
could
be
like
pretty
pervasive
throughout
the
codebase.
So
I
had
like
some
nitpicks
on,
unlike
my
like
some
of
my
code
and
some
of
its
calling
code
that
has
typos
in
it
like.
Do
you
guys
wait,
usually
for
a
merge
or
like?
Do
you
base
a
PR
on
top
of
your
existing
PR
and
like
haven't
emerged
like
have
them
block
each
other
or
lump.
A
That's
not
a
joke,
because
the
amount
of
time
that
is
expended
just
seeing
PRS
through
the
cycle
is
in
oral
argument
right
because
you
have
to
babysit
a
PR,
sometimes
to
get
it
through
the
testing
process
and
that
that
becomes
a
large
amount
of
time.
So,
if
you
have
typos
lump
them
together
into
functional
areas
like
if
you
have
typos,
they
pertain
to
cube
ATM.
A
A
You
can
lump
it
together,
but
just
you
know
try
to
make
things
atomic
where
possible.
So
this,
sometimes
the
random
walk
is
faster
than
the
straight
line
right.
So
if
you
can
break
up
any
large
PR
into
separate
atomic
things
that
can
be
merged
separately,
that
might
actually
recur
faster
I
have
your
PR
to
finish
reviewing
on
the
backlog,
but
there's
many
things
right.
Yeah.
G
A
Three
times,
alright,
thanks,
everybody
I
will
follow
on
with
updates,
and
next
week,
you'll
probably
see
a
bunch
of
modifications
to
the
backlog
for
Kubb
ADM.
So
if
you're
interested
in
contributing,
you
will
have
a
bunch
of
getting
started
PRS
as
well
as
these
much
more
thorny
or
ones
for
folks
to
talk
about
thanks,
everybody.