►
Description
Keynote: Kubernetes Federation - Kelsey Hightower, Google
About Kelsey Hightower
Kelsey Hightower has worn every hat possible throughout his career in tech, and enjoys leadership roles focused on making things happen and shipping software. Kelsey is a strong open source advocate focused on building simple tools that make people smile. When he is not slinging Go code, you can catch him giving technical workshops covering everything from programming to system administration. [@kelseyhightower]
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
But
it's
one
of
these
features
that
confuse
the
hell
out
of
everyone
every
once
in
a
while
new
technology
shows
up,
everyone
gets
super
excited
and
when
Federation
was
announced,
we
called
it
uber
Nettie's,
and
you
want
to
remember
that
that
project
name
and
everyone
lost
their
mind.
They're,
like
oh
we're
going
to
have
true
hybrid
cloud.
Many
people
still
use
a
term
hybrid
cloud.
It
means
absolutely
nothing
there's
no
such
thing
as
hybrid
cloud.
Either
your
clusters
in
two
places
or
one
place.
That's
all
it
is:
there's
no
hybrid
cloud
service.
A
A
A
A
A
Roughly
these
four
things
right,
you
want
to
be
able
to
securely
control
access
to
all
of
those
machines.
You
want
to
know
what's
running
on
them
availability,
so
things
that
run
across
those
machines.
You
want
them
to
actually
live
longer
than
one
of
those
machine
starts
to
die
and
then
resource
management.
Honestly,
most
people
don't
care
about
resource
management.
They
want
the
other
things
on
this
list,
but
resource
management
does
become
important.
A
How
many
people
are
using
like
a
cloud
provider
the
you
cringe
when
you
get
your
bill
right,
that
line
starts
to
matter
a
whole
bunch.
Once
you
get
that
person
voice
and
then
service
discovery
becomes
a
thing.
So
how
do
we
try
to
solve
this
before
we
got
into
this
world
of
kubernetes
and
user
service
systems,
most
people
in
the
real
world?
This
is
what
we
did
right.
Configuration
management
all
right,
so
we
took
all
of
those
machines
and
we
try
to
normalize
them
in
some
way.
A
The
idea
was
to
think
in
the
abstract
way
right.
We
can
define
what
should
be
on
those
nodes.
There
was
a
bit
of
a
problem
here
because,
usually
with
these
tools,
you're
still
defining
what
runs
on
the
node,
but
you
can
think
of
this
as
a
collection
of
servers
that
you
manage
is
a
single
logical
unit
under
your
configuration
management
system.
So
when
we
evaluate
what
we
got
from
configuration
management
systems,
I
think
we
fell
short.
We
got
two
out
of
the
four.
A
We
definitely
didn't
get
availability,
because
most
of
the
configuration
management
systems
pin
applications
to
a
node,
so
the
node
dies.
Your
config
management
like
to
bad
service
discovery
getting
this
because
I
was
too
low-level,
it's
really
replacing
our
batch
scripts.
So
we
want
something
else.
So
when
I
think
about
node
Federation
is
the
thing
you
use
every
day
in
kubernetes,
you
put
an
API
server
on
it.
Then
you
start
to
think
about
multiple
nodes,
like
a
single
machine,
run
this
application
somewhere
on
those
machines.
The
same
way,
you
think
about
a
multi-core
system.
A
A
So
now,
if
you
believe
in
node,
Federation
or
a
tool
like
ruminations,
it
exists,
then
I
think
the
same
situation
will
apply
when
you
have
multiple
clusters,
so
cluster
Federation,
if
we
keep
it
simple,
might
actually
solve
a
problem.
So
what
is
the
problem
we're
trying
to
solve?
We
introduced
the
second
cluster.
It
was
like
I.
Don't
need
node
Federation
for
that,
so
you
can
break
up
the
bash
scripts
and
do
for
loops
over
all
your
API
servers.
A
Where
you're
going
to
store
the
states
the
authority
place,
you
can
break
out
a
spreadsheet
and
decide
what
objects
belong
in.
What
cluster
and
over
time,
we're
going
to
invent
some
configuration
management
like
thing
for
kubernetes
cluster
is
inevitable.
We're
just
going
to
repeat
what
we
did
for
multiple
machines
from
multiple
clusters.
I
say
we
skip
that
part.
It's
going
to
be
painful.
Do
you
know
what
the
result
of
configuration
management
was
DevOps
good
therapy
for
inefficient
tools?
A
We
I
don't
want
to
know
what
DevOps
will
look
like
for
people
with
multiple
clusters
right.
Maybe
it's
called
sre
right,
there's
a
lot
that
goes
into
this
complexity.
So
now
that
we
have
control
of
this
system
with
a
great
API,
we
can
introduce
a
solution
on
top
that
solves
the
same
four
problems.
Okay,
so
how
do
we
do
this?
We
throw
another
API
server
on
top
now
the
way
you
should
think
about
this
API
server.
It's
not
the
way,
you
think
about
uber
Nettie's
of
solving
your
multi
cloud
problem.
A
This
is
meant
to
solve
your
multi
cluster
problem.
If
you
happen
to
have
clusters
running
in
multiple
clouds,
so
be
it,
but
that
has
nothing
to
do
with
Federation.
Now
having
data
from
two
separate
cloud
provider
or
a
single
API,
you
can
totally
go
build
some
custom
controller.
That
knows
how
to
migrate.
Apps
between
the
two
right
you're
going
to
have
to
specialize
when
you
start
to
think
about.
Amazon
has
an
elastic
block
device
and
Google
has
a
different
representation
of
that.
A
How
are
you
going
to
migrate
that
between
two
clusters,
the
truth
is
you're,
probably
not,
and
that's
where
you're
going
to
build
a
bunch
of
glue
code
in
the
form
of
controller?
So
we
do
this.
In
kubernetes,
today,
volume,
plugins,
Network,
plugins
I,
think
those
things
will
exist
at
a
higher
level,
but
we
shouldn't
try
to
bake
it
all
into
the
API
server.
We
got
in
trouble
with
this
with
kubernetes
today.
Anyone
ever
add
a
cloud
provider
to
kubernetes.
A
You
did.
Was
it
awesome
not
at
all.
It
wasn't
awesome
because
it's
all
baked
in
and
you
had
to
recompile
the
system
and
deploy
the
whole
thing
just
to
get
a
cloud
provider.
We
should
avoid
that
mistake.
In
Federation,
let's
just
treat
Federation
like
a
special
client
of
multiple
clusters:
okay,
and
if
we
do
that,
we
can
keep
it
simple.
So
what
happens
when
you
have
a
different
cluster?
Come
in,
maybe
you
have
a
bill
farm
and
you
have
to
make
a
decision.
A
Do
you
add
that
bill
farm
to
the
current
federated
control
plan
and
then
use
things
like
labels
and
policy
to
decide
that
bills
should
always
run
on
that
cluster?
Or
do
you
get
complicated
and
you
kind
of
have
this
overlapping
Federation,
where
one
of
the
other
clusters,
this
overflow,
for
when
your
bills
can't
fit
on
a
different
cluster?
Now
I'll
talk
to
a
lot
of
people
about
this
and
people
see
another
layer
on
top
of
this,
it
will
never
stop.
Well,
you
need
a
Federation
for
your
Federation
like
no
the
hell
you
don't.
A
We
need
to
keep
it
simple
and
reason
why
I'm
presenting
this
this
way
is
that
when
you
in
the
community
go
to
test
this
thing,
we
need
to
make
sure
we
have
the
right
frame
of
mind.
Is
this
a
coverage
client
that
makes
it
easy
for
you
to
build
tools
with
a
central
aggregation
point,
meaning
what
are
all
the
pods
running
in
all
of
my
clusters?
That's
a
great
question
for
the
federated
control
plane
to
me.
What
is
not
a
great
question
is:
could
you
migrate
me
from
a
sure
to
Google
cloud?
A
That's
what
you
should
probably
do,
even
though
I'm
not
biased,
but
it's
not
what
the
control
plane
should
be
doing.
Okay,
so
let's
keep
it
simple
and
avoid
this
complexity.
If
we
can
now,
when
do
you
use
Federation
I
think
when
you
have
an
application
that
spans
multiple
clusters
just
to
keep
it
super
simple,
so
you
have
to
availability
zones,
or
we
have
this
SLA
saying
that
we
will
only
scale
one
cluster
to
five
thousand
nodes
when
you
hit
that
limit
our
SLA
Zaroff
and
your
force
to
start
another
cluster.
A
If
you
want
to
scale
horizontally
so
in
that
world
you're
going
to
have
to
run
multiple
clusters
at
some
point
right
and
we
do
recommend
running
one
cluster
per
availability
zone
versus
trying
to
stretch
your
node
pool
across
multiple
availability
zones,
anyone
ever
lose
all
their
production
data
in
kubernetes.
Oh
that
guy
raised
his
hand
fast.
You
did
it
yesterday,
I'm
gonna,
ask
that
done
in
the
hotel
room
when
rehearsing
for
this
demo,
I
accidentally
called
the
delete
command
on
the
Federated
API
and
I.
A
Okay,
now
it's
fragile,
so
it
doesn't
work
is
not
my
fault!
Oh
one,
last
thing:
if
we
believe
that
the
Federation
should
be
this
client,
then
what
objects
should
is
support.
So
over
time
we
started
adding
objects
to
cover
nays
the
first
one
will
service.
You
know
global
service
discovery.
You
create
a
service
object
in
the
federated
control
plane.
We
push
it
down
to
all
the
clusters
and
we
figure
out
what's
available
and
then
we
can
create
DNS
entries
or
even
a
low
bouncer.
A
That
knows
how
to
send
traffic
across
all
those
clusters
and
then
one
that
four
we
sought
to
accelerate
our
pace.
You
know
things
like
events,
ingress
controllers,
namespaces
and
so
forth.
So
we're
getting
closer
to
that
and
then
we
added
a
few
more
objects,
but
then
what
about
the
ones
in
the
last
column?
How
would
you
fetter
eight
a
stateful
set
that
one's
hard
and
most
people
talk
about
it
like
a
TV?
All
you
do
is
replicate
the
data
all
right,
I
know
no
one.
A
So
all
of
these
things,
what
we
don't
want
to
do
is
add
them
to
Federation
and
introduce
new
semantic
meaning
because
that's
going
to
confuse
everyone
right.
So
we
have
this
list
of
things
that
need
to
go
in.
But
there's
a
discussion
required
what's
missing
from
the
discussion
is
more
people
who
have
hands-on
experience
with
Federation.
A
So
there's
a
repository
to
help
you
get
started
with
Federation,
but
we
need
more
feedback
before
we
make
crazy
decisions
on
that
last
call
because
it
will
be
bad
for
you,
so
I'm
going
to
do
a
quick
demo
all
right.
We
all
bow
our
heads
for
the
demo
gods,
because
if
it
doesn't
work,
it's
going
to
be
sad
all
right,
so
we're
going
to
do
is
we're
going
to
look
at
our
Federation
okay.
So
one
new
thing
that
you
get
with
the
Federation.
Is
this
a
list
of
cluster
objects?
A
So
at
this
point
you
should
list
out
clusters:
Wi-Fi
stopped
checking
your
email.
There
we
go
so
we
have
all
of
these
clusters,
so
we
have
five
of
them,
four
of
them
spread
across
the
globe,
and
the
idea
with
this
is
that
you
register
you
register
them
and
you
have
credentials
so
you
connect
to
it.
Just
like
a
client.
Now
we
only
support
a
subset
of
the
object.
So
the
first
thing
you
want
to
do
is
go
ahead
and
list
all
the
pods
across
your
Federation
and
it
doesn't
work.
Why
does
this
not
work?
A
This
would
be
the
first
thing
that
I
would
expect
to
work
so
right
now
in
Federation,
the
current
state
is
mostly
a
right-side
API.
You
write
things
into
it.
It
fans
it
out
to
all
the
clusters
and
you're
going
to
go.
If
you
want
to
query
this
system,
you
cannot.
So
what
do
you
do?
Bash?
Is
your
friend
you
get
your
for
loop.
No,
shame
all
right.
Let
me
show
you
how
that
looks.
You
do
that,
like
for
cluster
and
clusters
do
some
stuff.
A
It
will
list
all
the
pods
and
you're
winning
now
do
not
go
and
start
a
start-up
building
tools
for
this
I
know.
Some
of
you
are
anxious
to
get
going
on
that
funding.
Do
not
do
this
we're
going
to
fix
this,
but
the
current
state
of
the
world
is.
We
don't
have
much
on
that
so
on
and
said
in
the
car
like
dammit,
that
was
my
idea.
A
You
cannot
commercialize
a
batch
script.
Don't
try
that
so
we
don't
have
support
for
the
read
side
of
thing.
So
then
that
causes
tension
and
then
people
are
going
to
go
start
building
tools,
so
I'm
personally,
putting
a
lot
of
pressure
that
we
need
to
make
sure
that
anything
we
add
is
symmetric
if
I
can
add
a
deployment
that
I
need
to
be
able
to
list
the
pot,
and
if
I
can't
do
that,
it's
not
ready.
A
We
should
listen
in
any
of
those
columns,
okay,
so
the
next
thing
is:
what
is
the
magic,
so
the
magic
is
supposed
to
be.
I
can
give
this
thing
something
that
I
already
understand
here.
This
is
a
replication
set
running
engine
X
and,
ideally,
I
can
just
give
it
to
the
control
plane
and
by
default,
the
control
plane
is
going
to
take
80
of
those
and
spread
across
my
four
clusters.
Okay,
so
I
submit
that
and
then
they
start
running
and
then
I
can
also
expose
things
like
an
ingress
objects.
A
So
if
I
want
to
get
traffic
to
those
I
can
say
hey
cover
days,
I
want
to
use
a
particular
static,
IP
and
then
I
want
you
to
go
to
every
cluster.
This
is
a
service
discovery
thing
and
then
give
me
a
way
to
present
all
of
those
nodes
or
backends
into
a
single
ingress,
endpoint.
Okay.
Now
this
is
where
the
cloud
providers
tend
to
diverge.
Not
all
of
them
will
give
you
some
global
IP
that
you
can
wrap
around
all
of
your
clusters.
So
you've
got
to
be
careful
here.
A
What
this
means
so
I'm
going
to
show
you
what
happens
if
you
install
this
into
your
cluster.
So
the
goal
here
is
that
I
have
a
federated
set
of
clusters.
I
have
my
application
spread
across
all
those
clusters,
and
now
what
do
I
get
for
this?
What
you
should
end
up
with,
depending
on
your
implementation,
is
you
end
up
with
this
global
load
balancer?
So
here
I
have
this
global
load,
balancer
and
you'll
notice.
The
back
ends
here.
A
So
each
of
these
clusters
have
their
own
ingress
back
in
register
to
the
global
load
balancer,
and
what
I
want
to
do
here
is
use
geo,
ip's
or
you
know,
depending
on
where
the
client
is
coming
from
and
we
send
them
to
the
right
place.
So
we
need
to
generate
some
load.
Let's
look
at
the
IP
address
for
this
ingress
controller,
so
coos
ETL
gets
I
G
in
so
you
do
get
to
read
some
stuff
that
was
created
in
the
cluster.
A
Well,
I
spilled
that
wrong
all
right,
so
we
get
the
IP
of
this
thing
and
then
we'll
just
make
sure
that
it
works.
So
we
hit
this
all
right.
We
got
Engine
X,
but
we
need
a
little
bit
of
load
all
right.
So
what
I'm
going
to
do
is
log
into
different
machines
around
the
world
here,
I
have
something
in
South
Carolina.
This
is
a
state
in
the
US
and
what
I'm
going
to
do
is
have
it
ass
in
traffic
there's
going
to
go
slow,
long
requests
per.
Second,
don't
laugh
at
me.
A
That
is
scale
for
some
people,
they'll
be
jealous
all
right,
so
we
go
here,
and
what
we
want
to
expect
is
that
the
clients
in
North
America
should
flow
to
one
of
the
clusters
in
a
particular
region.
That's
close
to
it.
We
see
North
America
going
to
year.
We
know
something
is
wrong
in
one
of
those
clusters,
but
that's
what
we
want
by
design
so
we'll
let
that
run.
Let's
start
a
few
more,
let's
go
to
Japan
and
also
kick
this
off.
A
And
we'll
do
this
in
Oregon,
so
this
is
where
I
live
in
the
US
and
then
we'll
do
it
from
Belgium.
So
we'll
kick
this
off,
so
we
expect
now.
If
this
works
is
our
clients
should
start
flowing
to
the
appropriate
cluster
based
on
location,
so
we
have
Europe.
Now,
let's
see
if
we
can
get
all
four
streams
going.
A
A
All
right,
let's
turn
it
up
a
little
bit
I'm
becoming
impatient.
So
what
we
do
is
we
lift
the
cap.
We
get
serious.
You
know
what
we
say.
You
know
what
let's
just
do
like
100
per
second
or
let's
say
50
threads,
all
right
and
you
go
to
if
the
question
starts
falling
over,
make
sure
you
tweet
that
and
we'll
have
your
ass
off,
because
this
is
silly
you
do
what
you
rehearsed
all
right.
Let's
try
that
50
you
go.