►
Description
Meeting minutes https://docs.google.com/document/d/1ushaVqAKYnZ2VN_aa3GyKlS4kEd6bSug13xaXOakAQI/edit#
A
A
A
A
We
are
starting
looking
for
okay,
what
happens
whenever
I
want
to
upgrade
my
addons?
I
want
to
change
my
addons
and
along
this
wall.
This
way
I've
seen
other
requirements
like,
for
instance,
is
it
possible
to
inject
some
variable
from
my
clustering
to
my
addons
like
like
for
like,
for
instance,
it
is
possible
to
I
don't
know
to
inject
the
cluster
network
cedra
into
my
cni
config
and
so
on?
A
How
do
I
manage
upgrading
addons
when
my
cluster
gets
upgraded,
and
this
is
getting
really
an
odd
topic,
because,
especially
for
c
api
and
csi,
because
in
kubernetes,
csi
and
cpi
are
moving
out
of
three?
And
and
not
only?
We
have
to
manage
these
addons
in
in
conjunction
with
the
clustering
cycle.
But
we
have
also
to
help
the
user
to
transition
toward
the
out
of
three
csi
and
and
so
yeah.
B
A
C
Yeah
sure
I
can
go
first,
I
think
you
already
summarized
it
very
very
well.
To
be
honest
for
me,
the
only
the
only
requirement
that
I
have
about
this
is
to
be
able
to
update
those
resources
whenever
the
original
definition
in
the
management
cluster
has
been
updated.
So
whatever
was
tackled
by
my
original
design,
so
I
guess
I'm
gonna
be
here
more
as
a
listener
than
proposing
new
requirements,
obviously
that
that
said,
I'm
not
married
at
all.
C
With
that
proposal
that
I
wrote
I'm
more
than
happy
to
toss
that
and
then
come
up
with
something
something
new,
as
you
said,
something
maybe
more
holistic
that
that
fulfills
the
requirements
for
everyone
else,
but
yeah,
I
guess
that's
that's
my
two
cents,
which
is
mostly
I
don't.
I
don't
have
anything
else
to
out
there.
B
C
C
D
C
Comment,
one
comment
on
that.
I
think
it's
gonna
be
and
please
disagree
if
this
makes
sense.
But
I
think
it's
gonna
be
very
important.
Moving
forward
once
we
come
up
with
the
list
of
requirements
to
try
to
scope
down
what
are
the
things
that
we
consider
cluster
add-ons
that
need
to
be
tied
to
the
cluster
life
cycle,
which
is
what
copy
is
concerned
about
and
what
are
just
like.
C
It
is,
and
I
think
will
be
doing
us
a
bad
favor
if
we
start,
including
all
of
these
things,
and
we
try
to
to
include
all
those
use
cases
in
this
thing
which,
from
at
least
my
idea,
has
been
that
whatever
copy
allow
us
to
do
in
terms
of
like
crds
or
addons,
it
has
to
be
always
atoms
that
are
completely
tied
to
the
cluster
life
cycle.
Hence
they
can't
be
managed
in
any
other
way.
A
B
And
kind
of
so
in
csi
is
usually
you
have
totally
independent
versions,
and
you
might
need
a
newer
version
to
support
a
new
community
version,
but
for
cpi
it's
usually
that
each
of
those
cloud
providers
building
new
release
for
new
coordinated
series.
As
far
as
I'm
aware,
I
thought
in
openstack
in
aws.
C
B
E
So
I
can,
I
can
contribute
now,
if,
if
you
want
so
as
max
was
saying,
we
also
use
external
means
to
to
deploy
resources
and
clusters,
and
by
that
what
I
mean
if
we
use
pure
git
ops,
we
deploy
rct
and
we
point
our
city
to
our
repo
once
the
cluster
is
ready,
but
it
would
be
nice
to
have
something
to
deploy
the
necessary
things
to
have
rbcd
running.
So
all
you
see
is
right.
E
I
was
actually
going
to
mention
cnn
csi,
but
I
wouldn't
mind
if
there
was
an
option
also
to
deploy
a
physical
on
by
cargo
city.
That
is
then
going
to
point
somewhere
and
can
then
spin
up
all
their
meanings
right.
So
I
still,
we
still
need
to
bridge
the
gap
that
is
from
the
cluster
to
our
business.
Then
our
custody
can
do
the
rest
and
can
be
obviously
it
can
be
flux.
But
if
you
don't
have
anything
there
that
then
can
push
things.
You
still
have
to
bridge
that
gap
right.
E
E
A
What
I
go,
what
they
got
is
that
what
what
user
wants
is
that
they
want
to
manage
their
addons
with
a
single
tooling
and
for
some
addons
which
are
linked
to
the
cluster
they
they
want
to
cast
that
cluster
api
looks
in
into
their
dom
management,
but
not
replace
it,
and-
and
I
I
I
like
this
idea,
because
I
I
don't
think
that
as
a
cluster
api,
we
have
to
reinvent
adult
management.
There
are
new
great
tool
out
there.
There
is
a
helmet,
there
is
a
k-up
or
there
are
many
github
tools.
E
It's
not
just
sorry,
probably
it's
not
just
about
controls,
but
it
is
actually
about
ownership
of
the
cluster,
for
instance,
for
one
of
the
clients
I'm
using
cluster.
If
you
have
to
deploy
multiple
classes
that
they
are
controlled
by
different
teams
right,
so
you
actually
want
to
give
control
of
the
cluster
to
a
different
team
altogether
and
retain
the
control
of
the
control
cluster.
That
then
does
provisioning.
E
So
if
you
don't
necessarily
want
to
give
them
access
to
the
control
plasters
right,
the
main
cluster,
but
you
want
to
give
them
access
to
their
children's
and
different
people
can
have
access
to
different
classes
and
so
that
they
can
manage
their
classes.
In
fact,
it
would
be
amazing
if,
once
the
cluster
has
provisioned
the
cni
and
all
the
other
components,
it
could
give
ownership
to
the
children
cluster
right.
Achieving
class
can
take
control
of
those
even
right.
C
A
So
what
I
was
wondering
along
this
line
is
that
maybe
that
in
future
or
in
a
better
solution,
would
be
having
cluster
api
that
that
basically
orchestrate
something
else
so
or
adding
something
that
watch
cluster
api
and
orchestrate
something
like.
I
make
an
example.
If
we
have
a
controller
that
looks,
let's
assume
I
want
to
use
l,
but
I
think
the
same
apply
for
every
tooling
that
is
out
there.
So
if
I
ever
something
that
watch
for
cluster
api
and
maybe
in
a
nearest
future
in
for
runtime
extension
as
well.
A
We
can
leverage
on
templating
a
variable
substitution
stuff,
like
that
out
of
the
box,
and
the
same
should
apply.
My
cluster
is
upgrading
dearest
component.
The
system
accompanies,
should
I
have
all
the
knowledge
about
the
the
addons.
So
this
is
a
c
c
p.
I
I
does
mean
that
I
need
to
upgrade
it
when
a
kubernetes
version
is
upgraded,
or
this
is
a
cni.
A
E
And
and
for
rich,
are
you
thinking
this?
This
controller
is
going
to
run
on
the
management
cluster
or
in
the
created
clusters.
What's
your
is
that
for
how
do
you
describe
you
described
it?
I
assumed
it
was
in
the
control
class
in
the
in
the
control,
the
com
in
the
children
clusters
right,
that's
what
I
assume,
but
maybe
I
was
assuming
wrong.
E
A
Let
me
say
it
is
not
written
in
the
stone.
Someone
could
argue
that
I
don't
know
matrix
is
a
something
that
I
need
or,
or
someone
could
argue
not.
You
can
apply
matrix
totally
out
of
band
of
cluster
management,
so
it
it
is
something
that
we
have
to
to
refine,
but
I
think
that
what
we
have
to
ensure
is
that
those
are
cluster
add-ons
and
and
we
need
them
when
the
clusters
get
created,
upgraded,
updated
or
upgraded
and
deleted.
A
C
C
C
A
Yeah,
I
think,
yeah
we
we
are
on
the
same
page.
I
I'm
not
sure
that
the
hooks
will
be
enough
or
we
need
hooks
plus
controller
stuff
like
that
is
something
that
that
we
have
to
figure
it
out.
Probably
controller
is
something
that
we
we
can
have
we
can
implement
today.
Hooks
is
something
that
will
come
some
somewhere
down
the
line.
A
A
I
think
that
there
should
be
an
easy
part.
So
if
a
custom,
if
someone
comes-
and
it
does
not
have
flags
or
whatever,
we
should
give
him
instead,
the
cluster
is
that
they,
I
don't
know
cluster
api
addon
for
elm,
a
dawn
manager
for
helm
and
he
gets
a
flow
without
need
to
spinning
up
external
external
tooling.
E
There
is
another
model
that
I
that
I've
seen
recently
and
it
makes
a
lot
of
sense
to
me,
but
it's
gonna
get
more
more
embedded
into
the
the
management
cluster.
That
is
what
ranger
fleet
does.
I
don't
know
if
you
have
seen
the
renter
flick
project,
they
actually
have
bundles
defined
and
they
follow
this
model
of
of
a
management
cluster.
They
define
add-ons
in
there
and
then
you
can
manage
multiple
classes
from
the
management
cluster
and
they
solve
exactly
this
problem
right
installing
add-ons
and
you
can
have
a
look
at
it.
E
There's
a
rancher
fleet
and
they
actually
find
bundles
and
then
what
they
define.
They
define.
Cluster
groups
right
and
the
cluster
groups
have
specific
definition
configuration
for
the
add-ons,
so
they're
actually
exactly
trying
to
solve
these,
and
that
makes
a
lot
of
sense
to
me
because
what
they're
doing
they're
actually
defining
specific
configuration
for
a
group
of
clusters,
so
they
find
clustering
groups,
and
so
it
doesn't
matter
how
many
classes
you
have.
E
You
can
say:
okay,
this
is
the
bundle
for
discussion
groups
and
so
every
time
a
new
cluster
is
spinning
up
in
that
group,
it's
going
to
receive
the
same
amount
of
add-ons
applications
and
so
on
so
that's
a
model,
but
that's
they.
They
have
defined
a
new
customer
resource
definition
that
they
find
a
new
language
for
that
right.
So
it's
not
hooking
into
external
tools.
Then
it's
actually,
let's
say
adopting
what
the
similar
approach,
what
they
have.
B
I
think
the
interesting
point
about
that
is:
how
do
they
decide
when
to
deploy
which
version
of
things
so
when
we
upgrade
to
a
new
kubernetes
version?
How
could
that
I
looked
at
fleet
before,
because
I
tried
to
solve
that
multi-cluster
it
on
management
problem
in
the
past
yeah?
But
I
guess
it
probably
comes
down
to
similar
question.
How
do
we
bridge
that
gap?
Maybe
we
can
bridge
it?
For
both
I
mean
the
previous
idea
of
just
having
some
kind
of
bridge
to
hand,
but
also
from
looking
from
a
multi-cluster
add-on
management
perspective.
A
C
A
And
if
you
think
that
cluster
resource
set
at
the
end,
it
is
a
multi-cluster,
a
very
simple
multi-class,
so
it
is
something
that
we
already
have
in
the
radar,
because
we
look
at
the
cluster
from
a
management
cluster
point
of
view.
Where
you
can
manage
many
so
yeah
we
will
get
there.
I'm
more
concerned.
E
B
A
A
B
A
B
C
C
I
just
started
with
this
because
in
my
mind
it
was
the
path
of
less
resistance
to
get
to
something
that
was
working
right
until
now.
If
you
want
something
like
that,
you
need
to
implement
it
yourself.
If
now,
cluster
resource
sets
support
some
kind
of
update.
As
long
as
you
apply
those
new
cr
as
years.
At
the
same
time
that
you
apply
the
cluster
upgrade,
then
it
kind
of
works
regarding
what
you
said,
febreze
about
who's
interested,
I'm
interested
in
this
and
I'll
probably
have
some
time
to
work
on
this.
D
A
A
If
we
agree
that
this
direction,
my
concern
number
one
is.
That
is
something
that
we
need
now
badly
due
to
cpi
cni,
and
this
is
something
that
I
will
ask
to
the
community
tomorrow.
So,
and
that
means
we
will
gather
feedback
for
all
the
other
providers.
You
you
have
to
look
at
the
atmosphere,
but
we
need
feedback
from
all
the
other
providers.
A
A
To
the
community
in
terms
of
priority,
I
think
that
nevertheless,
it
is
a
good
idea
that
I
share
this
document
as
just
as
a
starting
point,
because
I've
invested
a
little
bit
of
time.
Writing
it.
And
then
we
we
try
to
make
this
a
proposal
with
the
goal
that
we
lay
down
a
path
forward
for
cluster
api,
orchestrating
advanced
tools.
C
Gotcha,
are
you
saying
used
to
clarify?
Are
you
saying
that
you'll
be
in
favor
of
if
the
community
is
in
favor
of
it,
adding
a
little
bit
of
functionality
to
cluster
resource
sets?
So
we
solve
the
problem
in
the
short
term
and
in
the
meantime,
we
start
working
on
that
helm,
slash
hook
whatever
document
as
our
long-term,
more
holistic
solution.
Is
that
what
you're
saying
yes,
okay,
yeah,
that
sounds
good.
I
I
will
agree
with
that
and
regarding
what
you
said
about
the
current
proposal
not
supporting
delete.
C
C
But
if
the
community
thinks
that
having
the
delayed
functionality
is
a
it's
a
road
blocker
and
without
it,
this
solution
doesn't
make
sense.
I
think
it's
something
that
that
design
could
be
expand
on
in
order
to
still
have
that
short
solution
that
fulfills
the
community
needs,
and
then
we
can
in
the
meantime,
move
forward
with
the
other
one.
E
I
was
just
asking
if
there
are
we
in
the
current
proposal
with
the
cluster
resource
set
to
to
upgrade
of
add-ons,
because
one
critical
things
that
I've
seen
in
the
past
is
that
you
may
have
a
critical
bug
in
one
of
the
components
and
if
you
cannot
upgrade
in
a
timely
manner,
that
can
be
a
real
issue.
Right,
so
great,
for
me
is,
is
more,
is
more
important
than
delete.
A
Yeah
current
proposal
is
about
upgrade
upgrade.
The
problem
is
that
it
is,
it
is
only
apply
a
new
yammer
and
if
the
delta,
between
the
old
and
the
new
is
the
object,
are
removed,
you
don't
get
what
expected,
because
it
will
be.
So
it
is
a
very
simple
upgrade
and
okay
making
it
more
complex.
Let's
mean
that
we
are
going
down
the
path
to
reinvent
elm,
and
so
this
is
where.
D
D
And
then
you
just
use
the
the
new
thing
when
it
comes
for
the
api
and
csi,
which
we
could
also
consider,
I
think
so
for
123
is
just
csi,
that's
actually
being
required
out
of
three,
but
we
could
also
consider
that,
like
to
just
put
documentation
up
because
csi
is
still
like.
I
mean
it
is
something
that's
required
for
a
cluster
for
storage,
but
it's
not
something
that
like.
If
you
don't
install
it,
the
cluster
would
never
come
up
like
cni
like
like
it's
not
gonna,
get
ready.
B
I'm
just
one
point
I
risked
before
in
the
chat:
it's
it's
definitely
not
great,
but
instead
of
implementing
an
upgrade
in
cluster
cuddle
in
cluster
resources,
you
could
also
just
deploy
another
cluster
resource
set.
I
mean
someone
has
to
do
that,
but
that
will
also
lead
to
an
apply
and
maybe
if
we
actually
want
to
have
another
solution,
maybe
documenting
that
thing
is
an.
C
Use
on
that
node,
I'm
not
sure
that
would
work,
because
if
you,
if
what
you're
trying
is
to
up
grade
or
update
a
resource
that
really
already
exists
in
the
in
the
workload
cluster,
even
if
you
create
a
new
cluster
resource
set
with
a
new
object
definition,
if
that
definition,
points
to
the
same
object
like
same
name,
same
name,
space,
same
type,
it
won't
reapply
it.
It
will
detect
that
it's
there
and
then
it
won't.
C
A
A
Assuming
that
we
have
to
manage
it
and
then
assuming
that
and
keep
it
for
granted
that
we
are
asking
to
the
user
to
manage
add-ons
in
in
two
different
way
or
if
we
want
to
have
a
unified
story
where
cluster
bi
just
orchestrates
our
other
stuff
and
and
so
the
user
is
happier
and
can
have
all
the
power
of
historians.
And
what
all
the
things
that
we
discussed.