►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Intro + Deep Dive: Multicluster SIG - Irfan Ur Rehman, Turbonomic, Paul Morie, Red Hat, & Quinton Hoole, FutureWei
SIG-Multicluster is focused on solving common challenges related to the management of many Kubernetes clusters, across multiple cloud providers (so-called hybrid cloud) and applications deployed across many clusters. In the introduction, we'll give attendees an overview of the current status of the multicluster problem space in Kubernetes and of the SIG. Beyond the technical details, we'll also cover the results of an on-going survey to gather feedback relevant to and assess adoption of the current SIG subprojects. Finally, we'll also be soliciting additional feedback from the community about the SIG's current efforts and future direction. This session will mostly be an open dialog; attendees will have the opportunity to ask questions in person as well as leave survey responses that the SIG will use to plan next steps.
https://sched.co/Uakw
A
B
A
Okay,
so
our
agenda
today
is
going
to
cover
a
history
lesson
of
history
of
cig
multi
cluster,
going
to
cover
the
current
cig
projects
that
we
have
and
also
sorry
I
am
figuring
out.
What
that
thing
is
I've
got
it
I'll
leave
I
won't
spoil
it,
though
exercise
for
the
reader
watcher,
so
we're
gonna
cover
current
sync
projects.
We
also
recently
in
cig,
multi
cluster,
have
had
a
survey
out
and
we're
gonna
talk
about
the
results
of
that
survey.
A
It's
around
the
adoption
and
interest
in
the
community
in
in
the
current
cig
projects,
and
then
we're
going
to
talk
about
what's
next.
So,
as
we
said
today,
is
supposed
to
really
be
an
interactive
discussion.
So
if
you
have
any
questions
or
comments
that
you
want
to
make,
please
do
feel
free
to
just
raise
your
hand,
and
someone
will
I
have
to
imagine,
take
like
a
triple
backflip
off
of
this
thing
and
go
and
give
you
the
mic.
So
you
can
ask
alright.
B
Okay,
so
we'll
start
off
with
a
little
bit
of
history,
so
the
efforts
around
sig
multi-class
sure
has
been
quite
quite
long.
The
initial
efforts
were
quite
under
kubernetes,
like
this
uber
control
plane,
which
overlooks,
or
over
arcs
all
the
kubernetes
cluster,
so
multiple
kubernetes
clusters
and
sort
of
solves
all
the
problems.
Whichever
might
be
there
when
multiple
kubernetes
clusters
are
considered.
The
initial
efforts
around
under
this
term
were
the
implementation
of
multi
zone
caters
clusters.
Okay
and
the
term
open
it
is
itself,
did
not
stick
for
long.
B
It
was
later
coined
into
Federation
and
Federation
was
viewed
as
this
API
surface
another
API
surface.
It's
like
the
same
gate
as
API
server
placed
on
top
of
the
existing
API
servers
and
given
special
meanings
to
the
resources
which
are
the
same
resources
that
cater
server
would
expose.
So
for
an
example,
our
deployment
when
passed
on
to
or
created
in
this
API
server,
which
we
call
federated
API
server,
the.
B
B
The
current
state
is
sort
of
beta
ready
for
cue
fed,
and
we
also
saw
another
effort
under
the
head
of
cluster
ID,
which
was
about
trying
to
fix
a
durable
ID
for
clusters
which
fall
under
this
multi
crystal
domain,
so
the
current
active
projects
or
the
current
projects,
some
of
them
are
not
active
right
now.
Our
cue
fed
cluster
registry
cluster
ID
was
an
effort
which
has
a
paper
and
currently
no
clear
implementation
and
the
Kuban
CI.
B
So
some
rough
idea
about
cue
fed
is
that
it
basically
distributes
the
configuration
or
K
same
KS
resources
across
multiple
clusters
and
the
term
or
the
definition
we
give
it
to
is
like
it
can
coordinate
the
configuration
across
multiple
clusters.
Its
current
state
is
just
short
of
beta,
and
that
is
something
that
we
basically
are
trying
to
define
if
we
should
take
it
to
beta
as
our
next
step
or
not.
B
We
have
seen
a
lot
of
activity
in
queue
fed
around
the
last
year
or
so
so
some
updates.
We
thought
that
it
it's
meaningful
to
give
out
some
updates
that
what
we
have
done
from
last
cube
corn.
So
from
last
cube
corn,
we
have
updated
the
propagation
status
so
earlier
it
was
difficult
to
find
if
the
propagation
of
resources
actually
happened
or
not.
So
any
problems
that
happen.
We
can
find
in
the
status
of
the
resources
which
we
call
that
proposition
status.
B
We
also
updated
how
queue
fed
handles
annotations.
So
we
found
scenarios
that
the
sync
reconciler
in
queue
fed
updated
annotations
or
we
pass
the
same
annotations
which
were
defined
in
the
Federation
well
plane
and
that
conflicted
with
the
local
K
tests
controllers,
because
the
controllers
would
also
update
the
annotations.
So
we
don't
just
update
them
anymore
and
we
skip
them.
B
Of
course,
we
can
override
this
behavior
using
some
other
mechanism.
We
change
the
API
group
from
q4k
tests
at
I/o
and
drop
that
name
caters
in
it,
because
this
does
not
necessarily
fall
under
the
core
gate.
As
a
pas,
we
also
made
it
possible
to
update
the
target
API
versions
of
the
resources.
So
earlier
we
always
used
to
target
to
the
latest
expose
version
from
the
gate
as
API
server.
Now
it
is
possible
to
update
the
versions
and
choose
whichever
we
want
to
target
from
the
control
plane.
A
So
the
next
project
we're
going
to
talk
about
briefly,
is
the
cluster
registry.
The
idea
is
for
it
to
be
provide
in
inventory
for
cluster
metadata.
So
what
are
the
endpoints
of
the
cluster
or
what
are
the
clusters
endpoints?
What
type
of
authors
accept
it's
currently
in
alpha,
and
this
is
one
of
the
areas
that
we've
been
trying
to
assess
the
adoption
of
the
cluster
registry
and
determine
what
criteria
there
may
be
for
moving
it
to
beta.
You
can
see
a
link
to
the
repo
there
on
the
bottom
of
the
slide.
A
One
of
the
the
ideas
that
the
sig
has
begun
to
discuss
in
the
last
months
is
a
concept
of
cluster
ID.
The
the
idea
is
that
DNS
names
come
and
go,
certs
come
and
go,
and
so
collectively
we
have
been
sort
of
identifying
some
properties
of
a
permanent
or
extremely
long-lived
ID
for
clusters
that
would
live
longer
than
a
cert
or
a
DNS
name.
It's
currently
existing
in
paper
form,
so
the
the
next
step
would
be
implementation
of
like
a
proof
of
concept.
C
All
right
guys,
so
eventually
we
had
a
couple
of
things
here.
We
were
trying
to
assess
state
right.
We're
trying
to
assess
whether
or
not
coop
fed
has
enough
adoption
for
us
to
be
ready.
You
know
and
feedback
to
be
ready
for
beta,
whether
or
not
there's
enough
interest
in
cluster
registry,
so
we
did
we'll
usually
do,
which
is
survey
a
bunch
of
people
now
at
this
at
the
point,
we're
actually
sort
of
compiling
results.
When
we
had
44
survey
responses,
it's
not
actually
enough.
C
I
think
there's
that
many
people
in
this
room
did
anybody
here
fill
out
the
survey
Wow
so
like.
If
all
of
you
filled
out
the
survey,
do
we
have
a
link
in
here?
Yes,
if
all
of
you
filled
out
the
survey,
then
we
could
double
the
number
of
responses
that
we
have
and
have
much
more
solid
information
about
a
bunch
of
these
things.
C
So
please
do
we've
got
the
link
there.
I
will
also
tweet
it
after
the
session.
You
know
to
again
get
get
the
the
link
to
that,
but
it
is
still
open
because
we're
still
trying
to
collect
more
information
about
people
with
multiple
clusters,
what
they're
using
what
they
want
everything
else.
So
let
me
give
a
few
highlights
from
the
responses
that
we've
received
so
far.
C
So
talk
a
little
bit
about
who's
already
filled
this
out.
We
had
a
good
mix
of
the
different
types
of
users
that
we
have,
including
both
end-users
and
consultants,
and
you
know
the
big
companies
and
providers
and
hosts.
We
have
said
a
couple
of
research
organizations
fill
in
the
survey
which
was
interesting
because
they
had
very
specific
needs
that
were
different
from
a
lot
of
the
production
users.
C
We
did
overwhelmingly
get
responses
by
people
who
are
using
kubernetes
in
production,
which
makes
sense
and
is
also
great,
the
and
a
mix
of
people
who
just
heard
over
the
survey
that
we
got
on
Twitter,
mostly
people
who
follow
the
say,
goo
at
least
read
our
meeting
minutes,
etc,
which
also
makes
sense.
We
also
provided
people.
If
you
look,
if
you
click
on
that
survey,
link
I
should
actually
have
it
at
the
bottom
of
each
page
here.
C
If
you
are
some
form
of
a
service
provider
or
you
know,
support
company,
you
can
fill
it
out
for
a
specific
customer
that
you
have
in
your
head
or
you
can
fill
it
out
for
your
customers
in
general.
Any
of
those
answers
are
perfectly
valid.
We're
just
trying
to
get
an
idea
of
what's
going
on
and
what
people's
needs
are
and
whether
or
not
you've
used,
say,
cube,
fed
and
how
it
worked
out
for
you.
C
Now
we
haven't
gotten
a
lot
of
responses
yet,
but
in
terms
of
getting
responses
from
the
right
people,
we
did
we've
done
pretty
well
between
the
44
respond
days.
There
are
7,000
and
some
clusters
under
management
size
varies
a
lot
across
you
know.
Most
people
are
running
somewhere
between
three
and
seven
clusters,
and
then
we
have
our
group
of
you
know
that
are
running
hundreds
to
thousands
of
clusters.
C
Now
one
of
these
want
to
discover,
you
know
come
through.
There
is
to
actually
figure
out
why
people
want
to
look
at
multiple
clusters
is
something
other
than
multiple
discrete
clusters
right.
Why
is
it
that
you
need
some
form
of
multi
cluster
functionality?
What's
it
for,
because
it
actually
turns
out
that
there,
no
theoretical,
multi
cluster
tools
can
actually
cover
all
use
cases,
so
we
need
to
get
an
idea
of
which
use
cases
are
more
popular
or
more
important.
C
The
number
one
thing
that
we've
heard
from
users
so
far
is
that
geo
distribution
of
some
kind
is
something
that
everybody
seems
to
have
in
common.
So
it's
out
of
people
here
are
running
multiple
clusters.
How
many
people
here
really
would
like
to
manage
a
distribution
better
yeah
VM?
So
this
wasn't
wasn't
a
big
surprise,
so
I'm
actually
going
to
ask
you
to
raise
hands
for
a
few
other
topics
too,
because
I'm
just
interesting
getting
this
right,
how
many
people
are
looking
at
multiple
clusters,
specifically
for
application,
high
availability
for
the
ability
to
failover
applications?
C
C
E
Sorry
nu
by
nu,
by
for
the
signal
to
cluster,
so
I,
want
to
understand
if
the
scope
of
the
multicolored
cluster
sig
is
just
managing
multiple
cluster
by
a
single
administrative
domain.
Or
do
you
also
cover
essentially
getting
multiple
independent
clusters
managed
by
different
users
the
different
it
means
into
a
more
cohesive
infrastructure,
because
that's
the
use
case
I'm
interested
in
right?
Well,.
C
A
One
of
the
things
that
that
we're
really
here
to
talk
about
today
is
like
understanding
what
the
scope
of
the
SIG's
should
be
when
we've
renamed
the
sig
from
sig
Federation
to
sig
multi
cluster.
We
picked
a
more
generalized
name,
but
we
really
didn't
establish
a
certain
mission
in
terms
of
what
the
SIG's
job
was
to
do.
F
C
C
A
That's
present
on
each
of
those
clusters,
so
the
idea
would
be
that
you'd
have
like
maybe
one
central
cluster
registry
and
somehow
kubernetes
happens,
and
that
that
the
cluster
registries
API
resources
are
present
then
on
each
of
the
clusters
in
that
registry,
for
like
a
centralized
telephone
book
or
cluster
location
is.
Is
that
a
use
case
that
resonates
with
anybody
here
today
if
I
could
see
a
show
of
hands?
C
Yeah,
okay,
so
it's
the
same
questions
about
tube
fed
who
fed
was
showing
more
actual
users
in
our
responses.
How
many?
How
many
people
here
use
coop,
fed
yeah,
see?
We've
got
yeah
the
end,
a
lot
of
our
written
feedback,
a
lot
of
the
freeform
responses?
Could
you
see
if
you
fill
out
the
survey
you'll
see?
C
A
C
A
Something
yes,
I
was
going
to
say
like
so
cute
fed
is
currently
in
this
state,
where
we
have
done
sort
of
beta
release
candidate
releases.
But
what
prevents
us
from
being
able
to
move
the
cluster.
Sorry
to
move
from
moving
cube
fed
to
beta
at
this
point
is
that
the
the
community
of
people
that
develop
it
is
just
simply
not
large
enough
for
us
to
responsibly
move
it
to
beta.
A
We
we
do
not
want
kubernetes
to
be
full
of
AP
is
that
are
at
a
beta
or
higher
level,
but
don't
have
enough
support
within
the
community
to
be
maintained.
So
one
of
the
things
that
we're
interested
in
understanding
is
whether
there
are
folks
that
are
interested
enough
in
it
that
haven't
previously
contributed
to
it.
That
would
be
interested
in
contributing
to
it
so
that
it
could
have
a
rich
enough
group
of
contributors
to
move
to
beta.
A
One
of
the
things
that
I'll
just
say
on
that
note
for
folks
that
are
here
today
are
white
might
be
watching.
The
recording
is
that
the
silver
lining
to
the
situation
that
we're
in
is
that
you,
you
can
go
and
try
cube,
fed
out
it's
basically
in
the
form
that
it
would
be
if
we
threw
the
switch
and
cut
Oh
dot,
one
dot
Oh
instead
of
another
RC,
so
you
can
try
it
out
in
the
form
that
it
would
initially
be
in
if
we
took
it
to
beta
right
now.
A
I
would
personally
love
to
hear
feedback
from
folks
that
maybe
haven't
used
it
yet
and
are
interested,
especially
in
terms
of.
Is
it
something
that
you
could
imagine
as
beta,
and
would
you
would
you
be
interested
enough
in
maintaining
it
are
interested
enough
in
it
having
a
long
lifetime
to
help
to
maintain
it?
So
let
me
let
me
ask
it:
is
there
anybody
in
that
category
here
that
maybe
you
you've
considered
using
it
but
you're
not
sure
about
using
it,
because
it's
in
alpha?
Do
we
have
anybody?
A
Are
you
interested
enough
in
using
it
that
you
would
help
maintain
it
if
it
moved
to
beta
here's
a
question
for
you,
I
see
some
shrugs
and
that's
okay.
Maintaining
maintaining
things
is
work
and
I.
Personally,
don't
want
to
do
any
work,
so
I
get
it.
That's
a
joke.
I
do
like
to
do
work
interesting
to
know.
Okay,.
G
G
We've
got
like
10
or
15
and
where
we
would
be
reasonable,
I
think
we're
thinking
it's
reasonable
to
expect
that
our
business
will
demand
us
to
make
50
a
hundred
by
the
end
of
this
coming
year.
I'd
have
to
upgrade
all
of
those
and
have
service
teams
who
are
gonna
onboard
to
them
all
over
the
place.
Part
of
that's
because
we
can't
make
clusters
that
we
think
might
fit
big
enough.
Maybe
and
some
of
it
is
you
know
to
your
location
and
all
that.
G
So
we
have
a
tool
right
now
that
allows
individual
teams
to
self-service
on
board,
and
then
they
just
we
just
let
them
use
coop
coop
CTL
from
there
I'm
imagining
some
of
them
will
think
that
that's
horrible
when
they
have
to
do
that
across
10
15
30
clusters,
depending
on
what
they're
doing
so.
If
there
is
some
Federation-
and
they
could
say,
do
this
and
that'll
go
over
there
and
I'm
done
I.
A
G
A
I
So
I
think
if
anybody
remembers
like
early
cue
cons
like
do
remember
what
the
number-one
competitor
to
curb
Randy's
was
when
we
the
same
survey.
It
was
not
May,
so
s--,
not
Cloud,
Foundry.
Sorry,
no,
not
swarm.
Shell
scripts
I
think
we're
at
that
same
phase
here
right
where
everybody
is
doing
their
own
bespoke
thing,
because
we
don't
have
a
thing
for
them
to
do.
J
K
I
K
I
K
Right
so
even
you're,
like
you're,
some
other,
your
your
scenario.
Over
there,
you
were
saying
you
you're
gonna,
explode
the
number
of
clusters
and
logging
in
and
trying
to
manage
those
individual.
Yes
totally
agree.
You
can't
do
that,
but
are
you
trying
to
manage
resources
across
all
those
clusters?
Is
that
what
you're
trying
to
do?
Are
you
trying
to
what
are
you
trying
to
do
across
all
those
clusters?
I
guess
is
really
the
question.
K
G
At
least
right
now,
so
we're
building
them
and
we
have
a
pipeline
to
upgrade
all
of
them
in
all
of
our
control
planes,
so
the
admin
services
edge
proxy.
Those
kinds
of
things
are
all
being
deployed
as
we
upgrade
the
clusters,
so
at
least
our
sent
our
management
stuff
is
going
out
pretty
easily
like
we
have
a
way
of
doing
that,
but
I'm
guessing
that
our
application
teams
who
are
fail
or
provisioning
their
own
namespaces
and
putting
them
wherever
they
want.
I
So
I
I
think
there's
a
couple
of
classes
of
problems
and
we
probably
ought
to
categorize
and
try
to
taxon
amaizing
to
these
cats
classes.
One
I
would
say
is:
there's
a
multi
cluster
set
of
problems
and
there's
a
cross
cluster
set
of
problems
like
multi
clusters.
More
like
fleet
management
right.
How
do
I
do?
How
do
I
upgrade
all
the
clusters
one
by
one
by
one?
I
How
do
I
see
what's
happening
and
then
they
and
the
cross
cluster
cases
like
what
I
want
to
do
when
I
have
to
bridge
between
these
clusters,
whether
that's
services
or
deployments
or
scaling
or
whatever
sort
of
semantically
sorry
network
network
I
mean
I'm
a
networking
person
at
this
point,
so
it
all
comes
down
to
networking,
often,
but
but
not
purely
actually.
There's
like
semantically
interesting
things
at
like
deployment
and
policy
and
and
control
levels
that
actually
are
about
cross
clusters,
not
just
multi
cluster.
So.
D
You're
right
on
I'm
I'm,
a
program
manager
for
a
sure
stack,
so
we're
doing
kubernetes
on
Azure
stack,
and
that
implies
folks
have
course
clusters
on
Azure
and
they
wanted
to
have
it
also
on
frame,
and
there
are
two
main
scenarios
that
we
deal
with
all
the
time.
One
is
solving
the
problem
about
fully
management.
That's
clear!
It's
well
understood.
People
have
a
set
of
coordinates
that
they
want
to
apply.
Is
a
single
organization.
H
Is
fleet
management
just
like
an
inventory
of
the
clusters,
or
is
it
managing
the
lifecycle
of
those
clusters,
or
is
it
like
managing
the
applications
like
this
is
the
like
breakdown.
This
really
need
right,
but,
like
fleet
management
means
something
entirely
different
to
everybody
in
this
room.
So.
L
I
D
The
priority
order,
given
what
our
conversations
with
customers
one
is:
I,
want
to
be
able
to
apply
the
same
policies
that
I'm
doing
on
this
cluster
across
the
20
other
clusters
in
an
EC,
so
governance.
The
second
one
is
managing
the
cluster
control
close
to
the
lifecycle,
so
I
am
able
to
deploy,
unable
to
upgrade
and
able
to
scale
so
that
will
be
about
the
priority
organ
of
those.
The
other
part
that
is
very
important
is
coming
up
all
the
time
which
is
morale.
D
It's
interesting,
but
and
people
just
about
everybody
wants
that
that
I
talked
to
is
I,
want
to
burst
out.
I
want
to
have
this
capability
of
having
replicas
across
clusters,
and
it
comes
up
all
the
time,
especially
I'm,
bragging
on
some
data
center
and
I'm
reaching
a
capacity
limit
now,
I
want
to
be
able
to
use
my
resources
that
I
pay
money
for
on
the
cloud.
A
So
a
couple
things
just
to
call
out
about
the
meta
of
what's
going
on
here
one.
This
is
the
kind
of
discussion
that
I
hope
that
we
would
have
here
to.
This
is
the
kind
of
discussion
that
I
hope
that
we
will
have
in
cig
multi
cluster,
so
I'm
really
happy
that
we're
having
it
here.
I
hope
that
we
can
exit
this
session
with
some
energy
from
the
folks
in
this
room
that
we
can
spend
discussing
this
in
cig
multi
cluster.
A
To
identify
you
know
what
what
is
the
problem
space,
because
we
really
have
to
map
it
out
before
we
can
know
what
to
attack
and
then
of
that
decomposed
problem
space
and
like
explicitly
listed
out
accounting
of
the
problem
space.
What
are
the
things
that
there
is
enough
interest
in
that
people
want
from
sig
multi
cluster
that
we
should
do
in
the
community,
so
I'd
be
curious.
Maybe
I
should
get
out
of
this
row
of
seats
here.
A
This
would
be
a
very
interactive
session
where
I
sit
next
to
you
and
talk
into
a
microphone
so
I.
Here's
a
question
posed
to
the
room
who
heard
something
in
the
last
five
minutes
that
they
want
to
get
from
kubernetes
that
they
want
to
get
from
the
community
because
there's
there's
things
floating
around
in
the
ecosystem.
But
do
you
want
to
get
it
from
a
vendor
or
do
you
want
to
have
a
basis
for
it
in
the
community?
I
see
hands.
So
that's
good
I
see
a
lot
of
hands.
I
I
H
Because
I
think
that's
what
everybody
wants
right,
I
mean
I
want
to
be
one
of
those
vendors
that
provides
it
to
you
right,
but
like
I'm,
assuming
most
people
like
there
are
things
that
I
don't
want
to
do
like
I,
don't
want
to
there's
plenty
of
open-source
databases
but
like
I,
don't
build
them.
I
want
a
vendor
to
give
them
to
me.
Well,
like
my
job,
is
to
like
help
our
customers
manage
lots
of
clusters,
but
I
don't
want
to
just
like
invent
something.
H
I
My
feeling
I'll
walk
over
toward
Steve
since
I
saw
Sam.
Let
me
just
let
me
say
one
thing:
I
agree.
My
feeling
is
what
we
want
to
do,
as
the
kubernetes
community
is
figure
out.
What
are
the
semantics
and
the
API
is
that
we
want
people
to
rely
on
such
that
the
vendors
can
go
off
and
implement
those
things.
I
This
is
what
customers
are
expecting
from
you
as
a
lowest
common
denominator,
and
this
is
what
like,
how
you
extend
to
beyond
that
in
a
sane
way,
and
then
let
the
vendors
go
crazy
and
I
think
this
is
what's
going
to
be
really
interesting
and
healthy
from
a
sort
of
Cambrian
explosion
of
ideas.
Mindset.
M
So
I
like
what
I
heard
I
thought
just
a
nuance
on.
That,
though,
is
you
know,
do
we
want
these
things
from
kubernetes
kubernetes
or
do
we
want
them
from
the
kubernetes
ecosystem?
Because
some
of
these
things
might
sit
around
and
all
set
outside
of
cube
and
orchestrate
them
definitely
feel
we
want
community
solutions,
but
maybe
not
directly
from
Cooper
kubernetes.
I
F
Let
me
it
kind
of
I
had
a
different
point.
I
think,
but
I
want
to
echo
on
or
build
on
that
too
right,
I
mean
I
heard
like
a
year
or
two
ago.
Thinking
people
say:
hey
Cooper
days
is
too
muddled
to
think
we
want
to
be
factoring
it
right.
So,
let's
start
doing
it
right
so
yeah.
That
makes
perfect
sense.
I
wanted
to
take
a
step
back
to
the
previous
conversation
like
what
problems
you're
trying
to
solve.
I
want
to
add
a
twist
here.
I
want
to
do
multi-tenancy
and
Multi
cluster
right.
A
G
Why
do
I
care
like
capacity
management,
is
one
of
those
as
we
have
a
lot
of
clusters,
and
we
want
to
make
sure
that
a
team
doesn't
overload
a
particular
cluster,
because
we
know
we'll
get
a
certain
size
and
then
it'll
tip
over
or
it'll
make
things
bad
for
them
and
all
of
the
same
kinds
of
things
for
deployments
and
scheduling.
Maybe
this
doesn't
meet
anybody
else's
model,
but
it,
but
it
sounds
like
kubernetes,
but
the
problem
is
that
we're,
instead
of
talking
about
a
node,
we're
talking
about
a
cluster
or
a
fragment
of
one.
H
Yeah,
like
I,
guess
it
doesn't.
Maybe
it's
just
the
way
that
we
use
multiple
lots
of
clusters
right
like,
and
this
isn't
us
as
a
cloud
provider.
This
is
us
of
like
how
we
run
our
micro
services
is
like
the
idea
that
there's
a
bunch
of
api's
that
we
interact
with
to
like
make
decisions
about
how
applications
get
distributed
to
clusters
just
didn't
work
in
our
mental
model.
For
some
reason
right.
We
were
much
more
concerned
with
like
distributing
applications
as
like
a
CD
process
and
using
api's
to
do
that.
F
Would
it
be
fair
to
say
Jake
that
the
dissonance
is
from
this
group?
We
got
the
idea
of,
like
ethically
in
the
taxonomy
right
we
hit
we're.
Trying
to
this
group
was
focusing
on
the
vision
of
like
one
consumer,
consuming
a
bunch
of
clusters
right
and
your
problem
is
providing
a
few
one
or
more
cluster
few
clusters
to
a
bunch
of
different
consumers
right
more.
The
fleet
name
problem
than
then
the
the
No
well.
A
I
Accurate
I
think
I
agree
completely.
I
I
was
you
know,
just
stick
you
lating
it.
You
know
you
couldn't
I
couldn't
do
it
couldn't
endorse
what
you
were
saying
hard
enough.
I
think
we
have
a
whole
range
of
individual
problems
that
we
tried
to
solve
with
one
solution
and
I.
Don't
think
that
that's
what's
gonna
work,
I
think
what
we're
trying
to
do.
You
know
I'm
at
Google,
for
anybody
doesn't
know.
We
know
we're.
We
hear
from
our
customers
all
the
time
I'm
using
two
clusters
and
boy.
I
Have
you
made
it
hard
right
like
like
really
and
and
honestly,
every
customer
of
any
value
to
a
vendor
is
going
to
have
more
than
one
cluster
right.
So
we
need
to
tackle
this
problem
and
if
we
all
do
it
in
individual
different
ways,
we're
gonna
just
hurt
our
customers.
We're
gonna,
completely
wipe
away
the
value
that
kubernetes
is
trying
to
bring
in
the
first
place.
So
now.
A
I
You
mentioned
specifically
deployment
and
I.
Think
deployment
is
a
really
interesting,
exceptionally
challenging
problem,
because
it's
super
opinionated
and
integrated
with
people's
development.
So
when
I'm
thinking
about
multi
cluster
I'm
actually
staying
away
from
deployment
right
now,
because
I
just
don't
think
we
know
enough
about
what
to
do
there
or
how
to
do
it
to
come
up
with
one
omnibus
answer.
I
do
think.
I've
heard
consistent
stories
from
users
about
things
like
services
and
policies
and
fleet
management
that
seem
much
more
approachable
from
a
like
making.
It
hurt
less
now.
A
One
of
the
things
that
is
a
civil,
so
silver
lining
or
an
asset
to
us,
is
that
we
have
these
projects
that
we
just
talked
through
earlier.
Maybe
I
should
also
stand
up
again.
We
have
these
projects
that
we
just
talked
through,
but
none
of
those
are
in
a
state
where,
like
there
is
a
backward
compatibility
guarantee
from
an
API
there's
still
basically
at
alpha
or
they're
on
papers,
so
we
have
an
opportunity
to
to
think
about
what
we
would
want
them
to
be
and
pick
them
apart.
K
K
The
reason
why
I
didn't
resonate
with
us
and
probably
with
other
vendors
as
well
I'm
sure
you
see
this
as
well-
is
that
we're
managing
lots
of
clusters,
but
they're
not
they're,
not
exactly
the
same
as
your
clusters
and
how
you're
building
your
apps,
because
we're
doing
a
very
large
number
very
in
many
cases,
are
very
homogenous
right:
we're
not
trying
to
distribute
your
applications
across
muscle.
That's
not
our
goal.
K
We're
just
trying
to
manage
many
many
clusters
for
the
purposes
that
you're
gonna
go
build,
really
awesome
applications
with
those
so
understanding
what
kind
of
problems
this
cig
is
trying
to
solve,
right
and
understanding
and
prioritizing
those
that
some
of
these
problems
are
even
a
simple
problem
of
understanding.
What
are
all
the
clusters
and,
what's
what's
running
on
them
consistently
across
the
globe.
It's
a
hard
problem,
it's
hard
to
do
consistent
and
even
able
to
answer
those
simple
questions.
K
A
Is
this
conversation
resonating
for
folks
that
are
in
this
room?
Can
I
get
a
show
of
hands
if,
like
you're
feeling
the
tuning
fork
go
like
this?
All
right,
I
see
hands,
that's
good,
so
how
many
of
you
will
will
show
up
at
the
next
signal
tea
cluster
to
talk
about
this,
because
that's
exactly
what
I
would
like
to
do
all
right.
A
That
would
so,
if
everybody
that
raised
their
hand
today
showed
up
at
the
next
signal,
Multi
cluster
I
believe
it
would
be
the
most
well
attended,
sig
Multi
cluster
meeting
ever
to
happen
by
two
hundred
three
hundred
percent.
So
I
hope
that
a
sizable
portion
of
you
do
show
up
to
our
next
next
meeting
and
I
would
love
to
have
this
discussion
to
continue
it.
Not
that
we're
ending
it
right
now,
but
I'd
love
to
continue
this
of
discussion
in
our
community
meetings.
A
The
first
step
is
that
you
turn
on
your
computer
now,
so
we
there
is
a
biweekly
meeting
it's
on
Tuesdays
lately
it
has
been
canceled
for
lack
of
agenda,
so
one
of
em,
and
we
can
enter
this
continuation
of
this
discussion
and
into
the
agenda,
but
this
actually
jogs
my
memory
on
another
point
that
I
wanted
to
make
just
show
of
hands.
Is
anybody
working
on
something
that
is
in
the
multi
cluster
functional
neighborhood
that
you'd
like
to
share
with
other
people
or
tell
them
about
show
of
hands
all
right,
I
see
some
hands.
A
So
to
answer
your
original
question,
there's
a
biweekly
meeting
it's
on
Tuesdays
in
kind
of
us
time
zone,
but
I
think
that
we,
we
certainly
could
be
flexible
on
accommodating
schedules
for
folks
that
might
be
in
a
pack
or
Europe.
So
we
can.
There
is
a
mailing
list.
Does
everybody
know
about
the
mailing
list?
Okay,
as
an
action
item
you
join
the
mailing
list.
I
will
publish
coordinates
for
the
meeting
to
the
mailing
list.
A
B
And
to
answer
that
particular
question
also
I,
like
earlier
Paul,
mentioned
that
we
have
a
good
amount
of
prior
art
and
we
have
done
some
implementations,
not
necessarily
framing
them.
For
example,
we
have
some
kind
of
service
discovery
that
we
implemented.
We
have
some
kind
of
distributing
resources
that
we
implemented
in
all.
B
It
need
not
be
used
in
the
same
form,
but
we
have
some
functionality
that
exists
and
I
I
saw
in
the
room
that
there
are
a
lot
of
folks
who
actually
are
either
interested
in
trying
it
or
already
trying
it
that
can
be
tried
out,
and
it's
not
necessarily
like.
We
are
not
necessarily
attached
to
that
particular
project
itself.
We
are
certainly
open
to
updating
it
or
changing
it
or
you
know
breaking
it
into
pieces
and
using
it
as
in
some
other
form
that
that
should
be
perfectly
fine.
In
my
opinion,.
M
If
I
Steve,
I'm
I
would
just
say
that
I've
been
floating
in
and
out
of
the
multi
cluster
sink
for
about
two
years
now
so
and
work
with
Paul
and
Ivan
and
other
folks,
like
I
just
say,
like
you
know,
the
SIG's
a
lot
of
just
RedHat
in
wall
way,
so
I'm
just
gonna
speak
from
a
RedHat
standpoint
that
we're
very
open
to
completely
redefining
this
thing.
I
think
you
can
see
from
attendance
like
I,
would
sort
of
characterize
our
trajectory
on
this.
M
I
I
There's
a
marked
uptick
in
interest
in
these
problems
right
like
we're
at
the
bottom
of
the
exponential
curve,
but
we
are
going
to
go
exponential
real
soon
and
every
customer
I'm
talking
to
has
these
problems
and
they're
feeling
it
more
acutely
now
than
they
ever
have
before
and
so
I
think
like
we
have
to
get
ahead
of
this
curve.
Like
you
know
now,
and
so
it's
not
that
I
think
there's
not
that
it's
been
lack
of
interest.
M
L
J
Hi
John
I'm
from
Walmart,
so
our
use
case
right
now
is
basically
in
the
multi
cluster.
We've
got
apps
that
need
to
be
distributed
out
to
all
Walmart's
same
version.
Same
functionality
needs
to
get
there
it's
stuff
like
checkout
or
inventory,
and
so
that's
like
1,500
stores
or
something
like
that,
all
going
out
with
the
same
sort
of
checkout,
app
same
versions
managing
that
app
lifecycle.
J
Different
versions
of
sort
of
different
overlapping
use
cases
are
different,
like
you
can
target
this
group
of
clusters
with
this
version,
but
in
the
same
group
of
clusters,
with
some
overlap,
comes
with
a
different
version
and
just
sort
of
managing
all
the
logging
I
feel
like
I'm
gonna
run
into
the
limits
of
like
a
status
sub
resource.
Really
quick,
so
I'm
wondering
what
y'all
are
sort
of
thinking
along
those
lines.
J
I
N
Yes,
so
I
just
wanted
to
add
a
more
and
at
that
point
didn't
know,
but
survey
will
fill
it
in
promise.
I'm
tour
from
ing
Europe
bank
very
different,
not
issues,
but
regulatory
issues,
of
course
determines
a
lot
what
we
do
and
we
don't
really
want
multiple
courses,
but
we
have
to
for
various
reasons
and
that
changes
how
we
view
this,
because
it's
something
we
need,
it's
not
something.
We
love
we're
not
against
it
either,
but
it's
just
it's
something
we
need
and
that
really
changes
how
we
have
a
look
at
this.
N
No
I
us
in
its
regulatory,
it's
different
uses
its
different
management
domains.
It's
it's!
It's
politics,
it's
no,
but
the
thing
is
that,
but
what
I
wanted
to
point
out
is
that,
from
an
organizational
point
of
view,
we
don't
particularly
want
us.
We
need
you,
but
from
a
personal
individual
point
of
view,
this
is
something
I
find
very
interesting
and
something
that
I
do
believe.
Oh
well,
this
is
going
to
change
a
lot
in
a
very
near
future,
and
this
is
something
we
as
a
community
really
need
to
to
get
homage
and
revolve
it.
So.
N
On
something
I
want
them
to
be
tightly
coupled
and
on
some
I
need
them
to
not
be
coupled
at
all.
So
therein
lies
the
problem,
but
it's
this
comes
down
to
that
in
our
aspect
of
the
package.
This
is
not
something
we
love.
This
is
not
something
we
cherish.
This
is
something
to
be.
We
just
need
to
do,
but
therefore
it's
not
in
our
case
not
gonna,
be
done
with
love,
I'm.
K
Wondering
everyone
doing
like
the
multi
cluster
and
you're
you're
getting
forced
into
it
right
to
some
degree.
Certainly,
yes,
yes,
growth
in
legal.
Whatever
the
case
is
you're
you're
dealing
with
multiple
clusters
is
some
of
the
problems
that
you
start
off.
You
learn
with
one
cluster.
You
grew
to
you
built
processes
and
tooling,
and
everything
across
in
that
cluster.
You
know
how
to
use
it.
It
worked
out
great,
and
then
you
tried
to
scale
it
and
that's
when
things
started
to
fall
apart.
Is
that
what
happened?
Did
people
just
kind
of
back
into
this
problem.
N
K
N
I
Like
in
abstract
space,
we
could
build
a
model
of
the
relationships
that
people
want
between
clusters
right,
whether
that
manifests
as
a
kubernetes
api
or
something
else
to
be
decided
right,
I
think
what
we're
missing
is
a
way
to
describe.
We
don't
have
words
to
model
this
relationship
right
right.
There
are
clearly
needs
for
clusters
that
are
highly
coupled.
We're
like
serviced
mesh
is
sort
of
the
model
of
what
people
are
thinking
of
how
they're
thinking
about
the
problem
and
there's
clearly
sort
of
fleet
management,
but
otherwise
don't
touch
them.
A
So
we
can,
we
can
do
a
couple
more
things
on
the
mic,
but
it
is
Thursday.
I
know,
folks
are
tired.
It's
been
a
long
week
at
cube.
Con
and
I
know.
Also
that,
like
there
is
a
submariner
talk,
that's
starting
in
a
few
minutes
that
probably
some
folks
want
to
go
to.
So
let's
make
it
last
couple
minutes
of
discussion
and
we'll
wind
this
down.
We
can
continue
this
discussion
in
the
next
signal
to
cluster
and
here
in
the
hallway,
sound
good.
E
So
wait
maybe
just
to
add
a
different
perspective
if
I'm
coming
from
the
science
community-
and
we
have
essentially
each
University-
wants
to
run
its
own
kubernetes
cluster,
but
then
we
want
to
aggregate
them
together
in
in
a
single
supercluster,
but
in
many
independent
that
overlays
of
different
clusters.
So
how
do
we
do
that?
We
have
done
similar
things
in
the
past
with
other
tools
like
batch
systems,
that
we
want
to
move
to
kubernetes
like
to
have
something
that
works
and
right
now
we
don't
know
how
to
do
it.
L
I
think
more
data
point
I'm
some
fun
I'm
from
Demonware
Activision
in
Pisa,
so
we
wrote
in
like
backends,
forgived
and
I.
Think,
like
a
lot
of
the
conversation,
so
far,
resonated
like
we
wrote
in
12,
plus
cluster
and
like
managing
them,
is
a
paid
I.
Guess
my
the
word
I
used
to
describe
them
would
be
fragmentation.
So
different
concerns
like
geo
is
one
limiting
the
blast.
Radius
is
another
one
and
so
on,
and
then
keeping
them
in
sync
across
these
dimension
has
been
a
fairly
tricky.