►
From YouTube: Cluster Federation in Kubernetes: Past, Present and the Future by Madhu C.S. & Quinton Hoole, Google
Description
Cluster Federation in Kubernetes: Past, Present and the Future - Madhu C.S. & Quinton Hoole, Google
In this session, I want to briefly present the current state of cluster federation in Kubernetes mainly focusing on what we aimed to accomplish, where we are today and where we want to go. After that I want to open the floor for discussion. The goal of the session is to discuss about the potential use-cases, challenges that people face while running cross-cluster workloads and the challenges that are hindering Kubernetes Cluster Federation adoption today. I would also like to listen to the feedback from our current users and hear about their experiences.
About Madhu C.S.
Madhu C.S. (madhusudancs@{github, slack, twitter}) is a Software Engineer on the Kubernetes team at Google where he works on Cluster Federation. Before Kubernetes, he worked on a number of different projects within Google Cloud. He also has a strong background in compilers and has worked on a JIT-compiler project at Google. Before joining Google full-time, Madhu helped the Google Summer of Code team for five years as an open source contributor. He is a long time open source contributor, a Gopher and a Pythonista.
About Quinton Hoole
Quinton is a senior engineer on the Kubernetes team at Google. Before joining Google he started EC2 at Amazon Web Services, leaving after 5 years to join Nimbula.com, another pioneering cloud computing startup, as senior engineer.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
B
A
A
You've
probably
heard
the
term
mentioned
in
numerous
Talk's
today,
hopefully
not
too
many
of
you
went
to
Brandon's
excellent
presentation
this
morning
or
demonstration,
there's
going
to
be
a
little
bit
of
overlap
between
what
Brandon
spoke
about
Morais
I'm,
going
to
be
speaking
about,
in
particular
module
demo,
but
with
no
further
ado.
Let's
get
going
so
we're
both
at
Google
at
the
moment
we're
active
contributors
to
the
cluster
Federation
project
as
I'll
discuss
later.
There
are
many
many
other
contributors.
I
don't
want
to
have
us
take
more
credit
than
is
due.
A
This
is
very
much
a
community
effort
and
I
guess.
Cluster
Federation
is
one
aspect
of
what
we're
going
to
be
talking
about.
The
other
part
of
it
is
is
building
global,
highly
available
multi
cloud
services,
and
this
is
a
huge
mouthful
but
but
I
guess
it's
important.
Multi
clouds
of
the
stuff
runs
across
different
cloud
providers.
You
can
run
clusters
in
different
public
clouds,
on-premise,
etc.
It's
highly
available.
You
can
handle
outages
of
clusters
and
its
global.
A
So
this
stuff
I
think
most
of
you
know
ready
Google
has
very
large
data
centers
so
to
many
other
people
these
days,
Amazon's
data
centers,
don't
look
too
different
than
this
either,
but
this
is
kind
of
stuff
that
you
all
know.
This
is
why
I
kubernetes
exists.
This
is
why
borer,
which
came
for
it
exists
because
managing
the
stuff,
with
anything
other
than
pretty
sophisticated
automation,
becomes
a
nightmare.
A
What
not
everybody
necessarily
thinks
about
at
first
pass
is
that
there
are
also
lots
of
these
data
centers
around
and
there
are
many.
These
are
Google's
data
centers
as
of
several
years
ago.
That
map
looks
different
now
than
then
it
does
on
the
screen,
but
the
Googlers
in
the
audience.
This
comes
from
our
public
website.
There's
not
internal
information,
but
you
know
Google's,
not
the
only
company
that
does
this
sort
of
scale
anymore,
Amazon's,
I,
think
AWS
has
got
35
odd
availability
zones.
A
A
So
what
if
we
took
that
principle
and
we
replace
the
things
on
the
right
hand,
side
with
entire
data
centers
and
what,
if
they
each
ran
in
different
cloud
environments,
the
classic
three
would
be
on
Google
on
Amazon
and
on
your
on-premise
data
centers.
The
model
looks
very
similar,
the
api's
that
are
used
to
the
clusters
of
just
the
good
old
kubernetes
api.
The
api
on
the
left-hand,
far
left-hand
side
of
that
diagram
is
also
actually
the
kuben
is
API.
A
So
all
the
concepts
that
you've
learned
all
of
the
tooling
that
you've
built
etc
speaks
the
same
language
to
the
pink
box
on
the
left-hand
side,
as
this
pink
box
on
the
left-hand
side
make
sense
so
far.
So
why
is
this
interesting
I
mean,
besides
being
an
interesting
academic
exercise?
Why
is
it
valuable
to
people
using
kubernetes
there?
There
are
a
whole
slew
of
reasons.
I
don't
want
to
spend
the
whole
afternoon
on
them,
but
I'll
just
whip
through
a
couple
quickly:
hi
availabilities.
Obviously
one
of
them
clusters
can
fail.
A
You
upgrade
clusters,
you
upgrade
your
software
in
them.
They
have.
You
know
acts
of
God
befall
upon
them,
I'm
sure
more
than
one
of
you
have
experienced
AWS
zone
outages,
for
example,
low
latency
tu
users
is
another
classic
example.
If
you
have
a
global
service,
you
want
to
serve
your
stuff,
typically
close
to
your
users.
A
A
You
can
subdivide
that
more
so
you
know,
Google
search
typically
would
not
serve
an
East
Coast
US
user
out
of
the
west
coast,
because
the
latency
would
be
noticeably
worse.
Policy
enforcement
and
auditing
will
go
back
to
this
diagram.
There
often
you
know,
company
policies
that
determine
can't
this
group
run
their
applications
on
AWS
or
GCE,
which
clusters
they
need
to
go
to
go
into.
Do
they
have
to
be
in
PCI
compliant
data.
Centers
is
the
sensitive
data.
Do
we
need
to?
A
If
you
have
an
orchestration
layer
to
help
you
do
that
scalability
a
lot
of
users
are,
you
know,
asking
questions
around
what
if
I
have
more
than
two
three
five
thousand
modes
that
I
need
to
orchestrate
my
containers
across
can
I
build
a
you
know
how
two
thousand
node
cluster
the
answer
is
it's
not
such
a
good
idea.
You
know,
even
if
everything
worked,
fine,
it's
just
generally
not
a
good
idea.
It's
not
something!
A
We
do
in
Google
very
commonly
it's
much
better
to
build
a
whole
bunch
of
you
know
multi
thousand
machine
clusters,
one
they
fail
independently
of
each
other.
So
you
don't
lose
a
hundred
thousand
machines
at
once,
and
there
are
various
other
reasons.
You
typically
don't
need
to
have
a
thousand
machines
near
one
geography.
You
typically
need
to
serve
more
more
than
one
geography.
We
have
that
number
of
machines
capacity,
overflow-
and
there
are
many
many
other
examples
of
why
this
is
useful
in
practice.
So
there
are
two
primary
building
blocks.
A
You
know
when
you
start
thinking
about
this,
a
little
bit
further
they're
two
primary
building
blocks
that
are
necessary
in
order
to
make
it
useful.
The
first
one
is
network
traffic
management
so
being
able
to
distribute
traffic
both
from
the
outside
from
the
Internet
to
your
to
the
correct
clusters
and
also
from
a
cluster
to
another
cluster,
and
so
that's
something
we're
going
to
be
talking
a
little
bit
about.
How
do
we
do
service
discovery
from
the
outside?
How
do
we
do
service
discovery
from
the
inside
of
a
cluster?
A
How
do
we
do
external
load,
balancing
geo
localities
or
ensuring
that
the
lowest
latency
access
to
clusters
from
users?
How
do
we
do
failover
if
clusters
go
down,
you
know
how
does
traffic
get
redirected
to
the
new
to
the
other
clusters
and
the
other
component
of
the
problem?
We're
solving
is
resource
placement.
So
where
do
we
put
all
of
the
load
balances?
Where
do
we
put
all
of
the
worker,
pods,
etc?
That's
it
behind
these
things.
A
How
do
we
allocate
IP
addresses
all
those
kinds
of
things
become
important
and
you
need
to
figure
out
how
to
do
that
sensibly?
Otherwise,
you
know
the
idea
is
not
very
useful
in
practice
all
right.
So
this
all
sounds
great.
Where
are
the
challenges
and
I
didn't
want
to
make
this
sound
like
this
was
all
roses,
and
most
of
you
will
probably
be
familiar
with
some
of
these
problems.
A
They're
different
different
bandwidth
charges
between
different
data,
centers
I've
tried
to
illustrate
that
in
the
diagram,
if
you're
going
between
inside
a
single
zone
and
a
single
cloud
provider,
typically,
all
network
bandwidth
is
free
and
your
latency
is
effectively
zero
as
soon
as
you
start
going
between
different
regions
between
different
cloud
providers
that
can
change
many
orders
of
magnitude.
There
different
service
discovery
techniques,
depending
on
which
cloud
you're
running
on
you
might
have
DNS
names
you
might
have
IP
addresses.
You
might
have
some
other
service
discovery
mechanism.
A
A
So
there
are
some
applications
that
are
more
easily
federated
than
others,
but
it
turns
out
that
the
class
of
applications
that
do
federated
well
is
substantially
large
and
interesting
enough
in
its
own
right,
past
present
and
future.
So
what
have
we
done?
What
have
we
built?
This
is
all
very
hand.
Wavy
requirements
stuff
up
to
now,
but
what
is
actually
there?
A
So
we
started
development
of
this
in
q2
of
this
year.
It's
important
to
note,
as
I
mentioned
earlier,
the
API
is
100%
kubernetes
compatible,
so
no
relearning
of
or
retooling
or
anything
else,
the
same
concepts
that
you
come
across
in
kubernetes
you'll
see
here
and
the
semantics
are
very,
very
similar.
A
You
can,
you
know,
take
your
existing
applications
and
deploy
them
with
their
current
configurations
directly
into
Federation,
and
you
know,
with
some
approximations
and
with
all
the
caveats
that
I
mentioned
on
the
previous
slides,
the
most
part
things
should
work
as
you
intend,
and
you
can
tune
from
that
point.
Q
to
2016.
We
built
the
basic
control
plan,
all
of
the
test
infrastructure
and
everything
else
that's
required
to
make
this
stuff
real.
We
also
built
federated
services,
which
we'll
be
demonstrating
in
a
few
minutes.
To
keep
you
all
excited,
slides
with.
No
demos
is
boring.
A
Federated
services
provide
you
with
multi
cloud,
multi-region
intelligent
service
discovery.
So
what
that
means
is
you
can
have
your
service
spread
across
AWS
GCE
on
premise,
if
you
like,
you
can
use
the
same
service
discovery,
you
can
look
up
the
same
DNS
name
wherever
you
are.
You
will
find
the
closest
shard
of
that
service.
If
you
have
one
in
your
cluster,
you
will
find
that
one.
If
there's
one
in
the
cluster
next
door
in
the
same
region,
you'll
find
that
one.
If
it's
nothing
available
there,
you
will
go
further
afield.
This
is
all
dynamic.
A
A
Q3
we
focused
on
you
know
we
sort
of
moved
from
a
build
the
basic
control
infrastructure
mode
to
let's
solve
some
really
interesting
problems.
We
released
federated
ingress
for
GCE,
so
you
have
a
global
l7
load
balancer
that
does
some
pretty
sophisticated
load
balancing
across
the
globe
based
on
a
single
IP.
So
there's
no
DNS
dependencies,
there's
no
TTL
x'
and
DNS
caching,
problems
associated
with
that.
We
also
did
replica
sets.
So
you
can
deploy
a
replica
set
that
spans
multiple
clusters.
We
do
dynamic
allocation
of
replicas
if
flusters
become
unavailable.
A
We
reschedule
the
replicas
into
other
clusters.
If
you
add
clusters
to
your
Federation,
you
can
up.
We
redistribute
the
replicas
that
you
have
across
the
new
clusters
secret
propagation,
so
a
pretty
common
use
case
is
needing
to
use
secrets
to
distribute
credentials
etc,
and
you
want
to
keep
these
consistent
as
soon
as
you
have
a
large
number
of
clusters,
you
have,
you
know
significant
configuration
problems.
This
makes
a
lot
simpler
and
events
just
make
it
easier
to
see.
A
A
We
build
some
tools
that
will
allow
you
to
deploy
a
federated
control,
plane
or
Federation
control,
plane,
easily
and
federate
clusters
into
that
with
single
line
control
commands,
we
also
build
federate
deployments,
so
those
of
you,
you
know,
could
Natty's
deployments
they
give
you
things
like
server-side,
rolling
upgrades,
etc.
These
are
implemented
in
Federation
as
well,
so
you
can
do
rolling
upgrades
across
multiple
clusters.
Daemon
sets.
These
are
things
that
run
on
all
the
nodes.
A
If
you
want
to
run
on
all
nodes
in
all
clusters,
you
can
do
that
with
a
single
department
daemon
set
at
the
moment,
jobs,
config,
Maps,
etc.
So
that's
a
little
bit
about
where
we
came
from,
where
we're
going
at
the
moment
and
in
the
future,
without
wanting
just
room
so
feedback
happening,
we
plan
to
spend
some
effort
on
the
following
areas.
In
the
coming
year,
policy
based
resource
placement,
so
something
that
is
quite
commonly
needed
is
the
ability
to
have
a
distinction
between
what
an
application
wants.
A
So
an
application
might
need
certain
kinds
of
nodes
or
certain
connectivity
to
other
parts
of
the
application,
but
a
company
or
some
some,
some
sort
of
other
party
other
than
the
application
deployer
might
have
another
set
of
policies,
which
is
we
preferentially
run
our
stuff
on
premise
and
when
we
run
out
of
capacity,
we
run
a
little
for
example.
So
we
like
to
overlay
that
policy
on
top
of
whatever
the
application
specified
policy
is,
and
that
should
ideally
be
something
that's
easy
to
declare
easy
to
understand.
A
These
are
the
people
specifying
these
policies
are
not
always
engineers
and
highly
technical
people,
so
this
should
be
easy
to
Express
and
easy
to
understand
improved
identity
and
access
management.
You
can
imagine
when
you
have
all
of
these
different
cloud
providers,
each
of
which
might
have
different
concepts
of
identity
and
you've
got
this
control
plane,
acting
on
behalf
of
potentially
multiple
users
against
different
clusters
and
different
cloud
providers,
things
can
get
complicated
pretty
quickly.
A
We've
got
a
very
simple
model
at
the
moment,
essentially
to
prove
out
the
concept,
but
there's
a
lot
more
work
to
be
done
there
to
make
sure
that
it
is
you
know,
solid
state,
flaps
and
federated
persistent
storage
doesn't
take
too
much
insight
to
realize
that
data
doesn't
magically
replicate
between
data
centers
or
cloud
providers,
so
in
the
same
way
that
kubernetes
pet
sets
or
I
guess
they're
called
stateful
sets.
Now
you
know
there
was
a
fair
amount
of
effort
put
into
making
sure
that
we
knew
how
to
handle
state
stateful
applications.
A
Well,
that
problem
is,
you
know.
Still
there
for
Federation
is
work
to
be
done
there.
We
have
some
pretty
good
ideas
of
possible
avenues
of
tackling
the
problem,
but
nothing
has
been
built
here,
hybrid,
federated,
in
gross,
as
I
mentioned
earlier.
The
ingress
that
we
built
in
version
1.4
is
based
on
Google's,
pretty
fancy,
l7
load,
balancing
infrastructure
and
not
everybody
has
access
to
that,
but
we
would
like
the
the
abstraction
of
federated
ingress
to
work,
irrespective
of
whether
you're
running
across
cloud
on
Google
or
not
so
that's
again,
we
have
a
design.
A
We
have
some
interim
at
least
some
initial
implementations,
and
we
have
to
roll
that
out
next
year,
private
federated
services
again
more
details.
If
you
need
them
available,
how
do
you
get
hold
of
us
as
I
mentioned
earlier?
This
is
very
much
not
a
Google,
only
effort.
This
has
been
an
effort
by
many
many
different
company
I,
just
called
out
the
main
contributors
on
the
right
hand
side,
but
there
are
several
others,
my
apologies,
if
anyone's
been
excluded,
huawei
core
OS,
Red
Hat
ourselves
at
Google
and
eBay,
have
been
just
some
of
the
major
contributors.
A
We
have
a
special
interest
group
that
contact
details
are
up
there.
Some
of
you
who
have
joined
it
there's
a
very
long
list
of
people
who
members
of
it
send
emails
read.
What's
going
on,
we
have
a
working
group
that
meets
every
Thursday
around
midday
Pacific
time.
You
can
join
that.
You
can
see
the
working
group
notes.
A
lot
of
people
have
come
to
me
since
we've.
A
Know
exactly
what
you're
doing,
because
we
read
the
notes
every
couple
of
weeks
and
we
we
know
what's
going
on,
even
though
I've
never
met
them
before
and
finally,
if
all
of
the
above
fails
quinton
mutter
there
are
email
addresses,
you
can
get
us
on
github,
you
can
get
us
on
email
and
now,
with
no
further
ado,
I
shall
hand
over
to
Madhu.
My
colleague
who
will
give
you
a
demo.
C
C
So
what's
the
problem
that
we
are
trying
to
solve
today,
I
want
to
build
a
global
service,
and
my
service
is
a
web
service
in
order
to
simplify
the
problem
for
this
demo.
I,
basically
just
have
an
engine
X
container
running
in
my
cluster
or
a
set
of
clusters
and
I
show
you
how
I
can
run
the
service
globally
across
multiple
clusters,
which
is
connected
to
a
federation
or
registered
with
Federation.
So
as
a
preparation
for
this
demo,
I
have
already
created
nine
gke
clusters.
C
There
are
nine
of
these,
but
in
this
resolution
it's
it's
really
hard
to
infer,
but
basically
there
are
nine
clusters
running
in
three
GCPD
regions
and
in
three
zones
in
each
of
these
three
regions.
They
are
all
running
one
one.
Four,
five
and
I
have
also
set
up
or
deployed
a
Federation
control
plane
on
one
of
these
clusters.
C
We
will
talk
about
the
Federation
control
plane
deployment
a
little
later
in
this
talk,
but
let's
assume
that
or
I
have
already
set
up
the
control,
plane
and
I
have
also
registered
these
clusters
with
Federation
Federation
is
only
responsible
for
managing
the
clusters,
that's
already
registered
with
it,
so
you
have
to
explicitly
register
or
D
register
clusters
from
r2
coming
to
Federation
or
with
Federation
or
deregister
clusters.
Out
of
it.
C
So
in
order
to
show
that
I
run
this
command,
you
use
the
same
Q
control
command
to
see
the
list
of
clusters
that
are
registered
with
Federation.
Also,
there
is
one
other
point
that
I
want
to
emphasize
here:
Q
control
can
talk
to
multiple
clusters
or
multiple
endpoints.
Basically,
it's
specified
through
the
Q
control
context.
My
current
context
is
a
federation
endpoint.
Once
you
have
your
Cube
control
talking
to
your
Federation
in
point
all
your
commands.
Look
very
similar.
I
just
need
to
get
clusters.
C
The
way
I
will
do
get
pods
or
get
deployments
just
to
show
that
I
show
you
what
my
current
context
is.
So
this
is
my
current
context.
It's
a
federation
cluster,
so
I
have
the
entire
setup.
Now.
What
I
want
to
do
in
my
service
is
I
want
to
display
the
basic
nginx
homepage,
but
in
addition
to
that,
I
want
to
show
the
region
or
the
zone
in
which
that
particular
page
is
running,
or
that
particular
instance
is
running
in
order
to
do
that.
I
have
a
shell
script
here.
C
Bottom
left
it
basically
contacts
the
GCP
metadata
server
to
fetch
the
current
zone
and
writes
the
zone
as
an
HTML
to
the
local
index,
dot
HTML
page
in
that
cluster.
It
just
does
that
this
is
a.
This
is
a
basic
shell
script
and
I
want
to
put
the
shell
script
onto
the
cluster's.
How
do
I
get
the
shell
script
onto
the
cluster's?
There
are
many
ways
to
do
it.
One
mechanism
to
use
is
a
config
map.
You
can
put
this
script
into
a
config
map
and
mount
it
on
a
file
system
in
your
cluster.
C
At
the
time
of
this
demo,
we
don't
have
a
federated
config
maps
right
now
we
do.
We
will
have
federated
config
maps
in
the
next
release,
which
is
in
about
a
month,
but
we
don't
have
it
right
now,
but
we
do
have
a
DirectX
secrets.
You
can
do
pretty
much
the
same
thing
with
secrets.
So
what
I'm
going
to
do
is
I'm
going
to
put
this
script
into
Federation,
which
replicates
to
all
the
underlying
clusters
to
all
the
nine
clusters.
So
left-bottom
is
my
script.
C
The
command
that
I'm
going
to
run
I
am
going
to
clear
this.
So
what
I'm
going
to
do
is
I'll
copy
paste
this
command.
It's
hard
to
remember
these
commands
during
a
demo,
so
so
I
just
created
a
federated
secret.
If
we
want
to
verify,
we
can
see
that
this
secret
is
in
Federation.
Now
we
can
do
a
Oyama.
C
To
see
the
details,
so
the
content
is
under
zone
fetch
that
Sh
key,
which
will
be
mounted
as
a
scrip.
The
the
actual
script
itself
is
basic
ste
4
encoded,
but
it's
there.
We
can
verify
it
if
you
want,
but
it's
not
really
necessary
right
now,
so
just
to
show
you
demonstrate
I
also
use
a
specific
context,
which
is
one
of
the
underlying
clusters
in
the
Federation.
So
this
is
the
cluster
and
we
have
the
zone
fix
secret
even
there.
So
it's
already
copied
to
all
the
underlying
clusters,
all
the
nine
clusters.
C
So
now
we
have
the
script.
We
want
to
run
the
script.
How
do
we
run
this
I'm
going
to
run
this
in
a
busybox
container
in
all
the
clusters,
but
we
don't
want
to
just
do
that.
We
also
want
to
display
the
page
written
out
by
the
script
as
an
engine
X
HTML
page,
so
I
have
a
replica
set
which
is
basically
a
federated
replica
set,
but
it
is
exactly
similar
or
API
compatible
with
the
kubernetes
api
replica
set
itself.
So
if
you
look
at
the
replicas
expect
it
has
API
version
kind
everything
the
same.
C
We
have
the
metadata
lib
name,
labels,
everything
and
I,
say
in
nine
replicas,
because
I
want
to
have
one
replica
in
each
underlying
clusters:
mine
cluster,
so
I
do
this
and
I
have
two
containers.
In
my
replica
side,
spec
one
is
a
basic
engine,
X
pot
of
nginx
container,
which
reads
from
ore,
which
has
the
dub
dub
dub
page,
and
there
is
a
second
container
which
is
just
a
busy
box
container,
and
it
runs
these
on
fetch,
sx
scribe
and
we
mount
this
script,
which
is
available
as
secret
in
each
underlying
cluster.
C
So
let's
submit
this
to
the
cluster
or
let's
submit
this
to
Federation
and
see
what
happens
before
doing
that.
I
want
to
take
a
step
back
and
set
a
watch
command
to
show
how
the
DNS
entries
work
acquaintance
already
talked
about
DNS
entries,
one
of
the
coolest
thing
about
Federation.
Is
we
just
don't
expose
the
service
internally
in
the
clusters?
We
also
set
up
DNS
entries
for
external
clients
for
clients
which
are
outside
the
clusters.
So
we
you.
We
have
two
DNS
provider
implementations
day.
C
One
is
Google
Cloud,
DNS
implementation
and
the
other
one
is
route.
53
implementation:
we
also
have
core
DNS
implementation
coming
up
in
the
next
Relays,
but
I'll
use
Google,
Cloud
DNS,
so
I
have
one
of
these
zones
created
in
my
cloud.
Dns
Google,
Cloud,
DNS
I
just
set
a
watch
on
it.
So
you
can
see
there
are
only
two
records,
the
namespace
record
and
the
SOA
record
no
records
or
cname
records
as
I
create
my
back-end
and
my
service.
You
see
this
getting
populated,
so
let
me
go
ahead
and
create
the
replica
set.
C
First
I
am
submitting
this
replica
set
to
the
Federation
endpoint,
so
the
replica
set
is
created
in
Federation.
It's
created.
The
message
says
so
no
need
to
verify
in
Federation,
but
in
the
meantime,
I
also
want
to
create
a
federated
service.
Look
at
the
bottom
left
is
this
thing
visible
to
all
of
you
in
the
bank:
okay,
so
federated
service
spec
looks
again
very
similar
to
or
exactly
identical
to
the
service
pack
in
kubernetes,
so
I
am
creating
an
engine
X
service
and
there
is
a
selector
port
and
whatnot
and
it's
a
pipe
load.
C
Create
the
federated
service:
let's
go
to
one
of
the
underlying
clusters,
any
arbitrary
cluster
and
verify
that
these
things
are
actually
created
in
the
underlying
clusters.
I'm
going
to
first
go
and
look
at,
you
can
already
see
that
the
top
right
is
getting
populated
with
records
cname
records
in
the
meantime,
I'll
also
show
you
whether
the
applicant
set
is
created
in
one
of
the
underlying
clusters.
So
we
asked
for
mine
replicas,
and
we
are
looking
at
one
of
the
clusters
here
which
is
running
in
Europe
West
1d.
C
C
So
Brandon
already
showed
you
the
browser
side
of
things.
It's
easy
to
just
use.
One
of
these
DNS
records.
That's
getting
created
here
on
the
right
side
and
go
to
a
browser
and
try
it,
but
because
of
DNS
cache
assigned
things
it
takes
a
while
for
these
things
to
propagate
so
I'm
not
going
to
do
it
right
now,
but
we
could
take
a
look
at
the
end.
Maybe
if
we
have
enough
time,
but
what
I'm
going
to
show
you
is
the
Intracoastal
traffic
or
the
inter
cluster
traffic
or
the
internal
clients.
C
If
you
have
a
service
or
if
you
have
a
micro
service
architecture,
it's
not
sufficient
that
your
service
is
working
from
outside
the
clusters.
You
also
want
your
services
to
work
from
inside
the
clusters.
So
if
you
have
your
service
running
in
cluster
a
you
can
access
your
service
from
cluster
B
because
of
Federation
I'm.
C
C
Federated
service
also
gives
you
a
internal
cube,
DNS
service
discovery,
name,
basically
a
DNS
name
and
the
DNS
name
is
very
similar.
If
you
look
look
at
the
well,
this
is
not
really
visible
for
people
in
the
back
I
guess
so
it
copy-paste
data,
one
of
the
shells
here
so
top
left.
If
you
look
at
it,
I'm
going
to
copy
paste
this
command,
which
is
basically
I'm
doing,
which
is
basically
aw
guide
but
I'm
W
getting
to
or
coming
to
one
of
these
sign
points,
which
is
a
service
discovery,
domain
name
within
the
clusters.
C
So
my
service
name
is
nginx
I'm
in
a
default
namespace,
the
next
part
is
or
the
the
next
in
fix
is
the
name
of
the
Federation.
We
have
a
concept
where
you
can
have
a
single
cluster.
Being
a
single
cluster
can
be
part
of
multiple
Federation.
So
you
can
specify
the
Federation
name,
and
if
you
do
this,
you
get
the
local
shard
of
the
service
running.
If
now,
I
do
have
a
local
shard
of
the
service
or
local
back
and
in
every
cluster,
so
I
am
getting
the
local
version.
C
If
I
did
not
have
a
local
shard
or
a
local
back-end
I
would
automatically
be
rerouted
to
one
of
the
closest
clusters.
This
is
the
power
of
federation
joins.
Thank
you,
I'll
hand
back
to
Quinton,
we'll
talk
a
little
bit
more
about
these
things
and
then
I
come
back
to
talk
about
the
Federation
control,
plane,
deployment.
A
Thanks
Badou,
hopefully
that
was
informative.
What
else
do
we
want
to
talk
about
so
some
of
the
stuff
that
you
saw
up
there
might
have
been
a
little
confusing
lots
of
consoles,
etc.
So
I
want
to
go
through.
You
know
what
was
happening
behind
the
hood
there.
So
we,
what
you
looked
at
previously
was
a
service.
We
can
also
do
the
same
thing
within
l7
load,
balancer,
ingress,
load,
balancer.
So,
essentially
what
happens?
Is
you
make
a
call
to
the
Federation
API
master?
A
You
specify
where
you
want
your
application
to
run
by
default.
You
can
leave
that
out
and
it'll
just
spread
it
across
all
the
clusters,
as
you
saw
in
the
previous
demo,
but
you
can
also
specify
which
clusters
you
wanted
to
run
in
and
what
the
weights
should
be
for
each
cluster,
a
single
call
that
gets
deployed
across
all
the
clusters.
This
example,
when
you
have
three
but
a
more
common
example,
is
you
know,
10
or
15
or
20?
A
You
create
the
service.
Similarly,
that
was
creating
all
the
clusters.
This
automatically,
in
the
case
of
a
global
ingress,
wires
up
an
external
global
load
balancer,
which
spreads
load
across
all
of
the
pods
in
in
your
service
across
all
of
the
clusters
and
upgrades
can
be
done
pretty
painlessly.
You
can
make
a
single
command
upgrade
to
upgrade
your
service
cross,
multiple
data,
centers,
l3e,
so
same
concept.
A
There
are
exactly
the
same
as
the
specifications
that
you
would
use
for,
kubernetes
control
play
and
make
sure
that
all
of
these
entities
are
created
across
all
clusters
and
all
the
DNS
management
is
managed
automatically
for
you,
its
failure
where
it
has
health
checks
against
all
of
the
pods
in
your
service
and
automatically
removes
and
adds
DNS
records
for
you,
based
on
which
pods
are
successfully
serving
at
the
time
which
data
centers
in
particular-
and
this
is
just
an
example.
If
things
go
wrong,
certain
cluster
goes
down,
fails
automatically.
All
the
DNS
stuff
gets
updated.
A
It's
a
good
question.
The
answer
is,
it
looks
up
the
same
thing
so
so
it
looks
up
a
name
which
is
the
name
of
the
service
that
at
once
and
the
name
of
the
Federation,
so
you
can
either
specify
I
only
want
to
find
that
service.
If
it's
in
my
cluster
or
the
lookup
should
fail-
or
you
can
say,
I
would
like
to
find
it
in
my
cluster
if
it
exists
or
otherwise
give
me
the
closest
replica.
A
Yes,
correct
for
the
load
balancer
that
exposes
that
service
externally.
Yes,
yes,
so
the
current
incarnation
of
this
provides
public
services,
so
the
services
are
exposed
on
load-balanced,
external
IPs
and
publicly
visible
dns
names,
as
you
might
have
mentioned,
I
might
have
noticed
on
the
earlier
slides.
Creating
private
services
is
definitely
a
user
requirement
and
we'll
work
on
that
in
the
coming
future,
so
where
the
DNS
names
are
internal
to
the
clusters
and
the
IP
addresses
are
not
visible
on
the
public
internet.
So
that
answer
your
question
excellent.
A
C
Right
so
there's
a
hint
right
here:
good
Federation,
control
plane
is
another
application
in
kubernetes.
So
we
started
working
with
this
assumption
that
we
already
have
kubernetes
cluster
as
why
not
leverage
all
the
features
that's
available,
so
it
runs
as
an
application
in
one
of
the
underlying
clusters.
Let's
take
a
look
at
how
that
actually
works
or
the
architecture.
C
So
if
you
want
to
create
a
Federation,
you
want
multiple
clusters.
That's
exactly
why
you're
creating
Federation!
It's
not
really
necessary
that
you
create
all
your
clusters
up
front.
You
can
create
the
clusters
or
provision
the
clusters
at
any
time
that
you
want
decommission
them,
join
with
Federation
unjoin
and
whatever
and
whatnot
every
possible
permutation
and
combination.
But
you
need
at
least
a
single
cluster
to
set
up
Federation
control,
plane
itself,
which
we
call
as
the
host
cluster.
It
doesn't
matter
which
cluster
is
your
host
cluster?
C
In
my
example,
here
I'll
choose
cluster
2
as
my
host
cluster
and
I
have
three
other
clusters
already
set
up
for
me,
but
you
can
add
clusters
as
you
want,
so
a
federation
for
people
who
are
familiar
with
the
kubernetes
control
plane
itself.
It
looks
exactly
identical
or
almost
identical.
We
have
an
API
server,
Federation
API
server
like
the
kubernetes
api
server
and
that
server
the
server's
endpoint
has
to
be
exposed
to
the
outside
world
for
people
to
make
API
requests.
C
So
we
deploy
the
API
server
itself
in
a
part,
but
we
use
the
kubernetes
deployment
resource
or
the
API
object.
So
the
Federation
API
server
is
a
deployment
and
we
expose
that
using
the
kubernetes
api
service,
IP
service
resource
and
we
have
a
federation
controller
manager
and
that
again
is
deployed
as
another
kubernetes
deployment.
C
We
need
a
data
store
to
store
the
state
of
the
API
server
like
kubernetes,
so
we
have
HCD,
which
backs
the
Federation
API
server
state
itself
or
the
storage
itself,
and
then
we
need
to
set
up
all
the
credential
certificates
and
all
those
things
and
configure
DNS
dns
providers,
cloud
providers
and
whatnot.
All
these
things
can
be
done
manually
and
in
my
setup,
I'm
using
cluster
2
as
an
example,
but
again
I
want
to
re-emphasize
it
doesn't
matter.
Also,
you
can
spread
your
control
plane
across
multiple
clusters.
The
current
scripts.
C
Actually,
don't
do
it,
but
we
have
a
plan
on
making
the
Federation
control
plane
itself
ha1
of
the
future
releases,
not
the
one
coming
up
in
December,
but
in
a
future
release,
but
you
can
do
all
these
things
by
hand,
but
the
problem
is
these.
Things
are
a
little
error-prone,
but
let's
talk
about
that
in
the
next
slide.
So
once
you
have
the
control
planes
set
up,
you
want
to
register
already
register
clusters
in
order
to
register
a
cluster.
C
You
basically
send
this
a
que
request
this
a
PA
if
this
object
to
the
Federation
API
server
endpoint.
This
is
probably
the
only
special
APA
resource
that
we
have
in
Federation
today,
the
API
to
register
our
already
registered
clusters.
So
you
specify
the
cluster
name
and
the
API
server
endpoint
of
the
kubernetes
cluster
and
a
reference
to
a
secret
which
contains
the
credentials
to
access
that
cluster,
and
you
submit
this
to
your
Federation
API,
server,
endpoint
and
boom.
Your
cluster
is
registered,
cluster
is
registered
and
you
can
start
sending
API
requests
to
that.
Again.
C
All
these
things
can
be
done
manually
and
it
many
people
actually
do
this
manually.
If
you
go
look
at
Kelsey's
tutorial,
he
has
all
the
steps
listed
to
do
these
things
manually,
but
it
is
very
error-prone.
So,
in
order
to
simplify
these
things,
we
are
building
a
new
command-line
tool.
Called
s
cube,
fed.
It
stands
for
kubernetes
federate.
C
It
reduces
both
the
control,
plane,
deployment
itself
and
cluster
registration
and
D
registration
to
two
commands
cube
fit
in
it
will
deploy
the
Federation
control
plane
for
you
in
a
single
command
and
cupid
join
will
register
your
cluster
with
Federation
again
in
another
single
command.
There
is
an
equivalent
cupid
unjoin,
but
it's
not
listed
here
anyway.
This
is
how
the
control
plane
works
and
that's
it.
We
yeah.
B
A
Absolutely
so
the
security
story
here
is
pretty
weak
at
the
moment.
You
know
we
explicitly
decided
to
do
that,
just
to
prove
out
the
concept
and
get
people
to
use
this
stuff
in
a
sort
of
proof
of
concept
context.
The
model
is
fairly
simple.
At
the
moment,
a
user
provides
the
credentials
that
they
would
use
to
access
a
cluster
to
the
Federation
and
the
Federation
act
on
their
behalf.
Using
those
exact
credentials,
it's
not
secure.
It's
not
multi-tenant.
A
It's
it's
not
the
right
way
to
do
things,
but
it
does
enable
you
to
you
know
at
least
play
around
with
the
stuff.
The
model
going
forward
is
going
to
be
much
more
sophisticated
than
that.
Essentially,
the
Federation
will
impersonate.
Particular
you
know
more
than
one
different
user
and
use
their
each
of
those
credentials
which
it
will
get
from
an
external
source
to
operate
on
behalf
of
the
user,
explicitly
as
opposed
to
pretend
they
are
at
that
person.
Yeah.
A
A
B
A
Filtering
I
have
not
applied
my
mind
to
that
problem.
I
must
confess
the
general
principle
is
that
what
we
do
is
we
work
through
the
kubernetes
api
and
we've
been
adopting.
You
know
the
most
important
things
first,
but
yes,
anything
that
kubernetes
supports
that
you
could
reasonably
expect
to
be
implemented
in
the
Federation
control.
Plane
would
be
in
on
the
roadmap
exactly
how
we
do
that.
It's
not
clear
to
me
yet
yeah.
B
B
A
Sure
that
that's
a
good
question,
so
the
model
is
similar
to
kubernetes.
So
at
the
at
the
low
end
of
the
spectrum,
you
have
a
single
API
server.
You
have
a
single
controller
manager
and
a
single
sed
server,
and
these
things
get
automatically
restarted,
so
they're
self-healing,
but
not
highly
available
in
the
sense
that,
if
any
one
of
them
fails,
you
will
have
an
outage
for
the
period
of
time
that
it
takes
to
bring
up
a
new
pot.
A
The
sed
server
is
typically
backed
by
a
persistent
disk
which
outlives
particular
nodes,
so
it
comes
back
again.
Reattaches
the
same
disk
and
life
is
good
within
a
few
seconds.
The
next
step,
up
from
that
is
to
have
multiple
API
servers.
You
can
have
as
many
as
you
like
behind
the
service.
You
know
you
can
send
your
request
through
any
one
of
them.
Those
are
all
backed
by
a
potentially
or
typically
a
quorum
of
SED
hosts.
A
So
you
have
commonly
three
or
five
of
these
things
and
in
the
case
of
three
you
can
lose
one
and
have
it
replaced
and
you
have
no
outage.
You
have
a
master
elected
controller
manager,
which
does
things
are
scheduling,
etc.
So
you
kind
of
multiple
of
those,
but
you
can
have
a
leader
election,
so
you
can
have
multiple
running
at
the
same
time
as
soon
as
one
dies,
the
other
one
takes
over
as
leader
and
then
the
extension
that
applies
in
the
Federation
cases.
A
You
can
spread
these
things
across
multiple
clusters,
so
a
common
h.a
deployment
scenario
would
be
multiple
clusters
in
the
same
region
in
the
same
cloud
provider,
so
low
latency,
relatively
low
latency
network
connectivity,
low
cost,
but
three
replicas
in
different
availability
zones.
You
can
lose
an
entire
availability
zone
and
you
will
have
no
outage
of
your
control
plane
and
the
same
goes
for
the
API
servers,
multiple
in
different
zones
and
controller
manager,
which
does
leader
election
across
URLs.
C
A
C
If
you
set
up
a
federated
service,
you
have
your
DNS
entry
configured
to
talk
to
all
the
services.
You
just
talk
to
the
one
that's
closest
to
you
and
if
that
goes
down
you
dock,
your,
your
DNS
traffic
is
automatically.
When
you
do
a
DNS
query.
You
automatically
get
back
I
DIF
different
data
card,
and
you
start
talking
to
that
and.
A
One
thing
is
with
bringing
up
here:
a
lot
of
people
get
worried
about
the
resiliency
of
the
control
plane.
It's
important
to
note
that
the
failure
of
the
Federation
control
plane
does
not
result
in
the
failure
of
all
the
clusters.
It's
completely
independent.
The
clusters
carry
on
running
woopsie
and
the
clusters
carry
on
running.
You
know.
Some
of
the
DNS
updates
will
will
not
happen
while
the
Federation
control
plane
is
down,
but
everything
else
works.
Fine,
the
load
balancers
that
load
balanced
onto
all
of
those
things
are
configured.
Everything
works.
The
replicas.
A
The
replica
set
controllers
in
the
underlying
clusters
are
making
sure
that
the
right
number
of
pods
are
running,
etc.
So
you
know
it's
it's
not
such
a
catastrophe
to
have
your
entire
Federation
control
plane
go
down
other
than
the
fact
that
you
lose
API
access
to
it,
but
your
application
carries
on
running
largely
unaffected
and
the
same
goes
for
individual
clusters.
By
the
way,
if
you
lose
an
individual
cluster
in
the
Federation,
you
know
the
services
generally
carry
on
running.
Yes,.
B
A
That's
absolutely
fine,
and
what
will
happen
is
if
you
have
an
outage,
depending
on
how
you've
deployed
your
kubernetes
clusters,
you
could
do
that
upgrade
in
a
in
a
way
in
which
your
API
remained
available,
in
which
case
the
Federation
wouldn't
care.
It
would
keep
communicating.
We
have
API
backwards
compatibility,
so
everything
should
work.
Fine,
if
you
have
an
upgrade
scenario
where
the
API
server
goes
down
for
some
period
of
time,
the
Federation
will
see
that
we'll
mark
it.
A
As
you
noticed
when
we
listed
the
clusters
earlier
on,
you
had
ready,
so
your
cluster
will
become
unready
for
some
period
of
time
and
then,
depending
on
how
your
deployments
etc
are
set
up.
It
could
actually
move
replicas
to
a
different
cluster
for
you
and
that's
all
optional,
so
you
can
either
configure
it
in
such
a
way
that
the
cluster
will
remain
as
it
is,
or
that
it
that
the
Federation
control
plane
will
redeploy
the
replicas
into
other
clusters.
For
you.
A
A
little
bit
outside
of
the
description
we've
run
out
of
time,
but
I'm
happy
to
talk
about
it.
Afterwards,
we
do
have
you
know
full
backwards.
Compatibility
of
API
of
API,
so
so
a
client
talking
to
a
cluster
can
assume
that
it
is
talking
to
a
version
1.4
cluster
or
even
if
it
is
talking
to
one
point,
I
plus
1
yeah,.