►
From YouTube: Self-hosted, Scale, and Federation with Kubernetes v1.4 and Beyond by Brandon Philips, CoreOS, Inc.
Description
Self-hosted, Scale, and Federation with Kubernetes v1.4 and Beyond - Brandon Philips, CoreOS, Inc.
"Cloud native architecture is becoming more flexible and accommodating of the increasingly diverse needs of operators. This talk will provide an overview of the latest developments on the open source projects CoreOS develops and contributes to in the Kubernetes ecosystem. This talk will cover the introduction of cluster federation, the status of easy to operate ""self-hosted"" clusters, and the latest progress on etcd v3, and why all of these developments are pushing the envelope of hyperscale infrastructure. And discuss the user demands that are driving these changes.
With the rise in demand to run internet services, businesses are all becoming software companies. For businesses to scale and meet these demands, automatic, painless software updates and improved infrastructure are essential. Cloud native infrastructure is designed to meet these demands, and CoreOS is making cloud native solutions available for everyone, with what we call GIFEE: Google’s infrastructure for everyone. This talk will cover the major investments in open source technology CoreOS is making to enable GIFEE."
About Brandon
Brandon Philips is helping to build modern Linux server infrastructure at CoreOS as CTO. Prior to CoreOS, he worked at Rackspace hacking on cloud monitoring and was a Linux kernel developer at SUSE. As a graduate of Oregon State's Open Source Lab he is passionate about open source technologies.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Well,
Wow!
Well
thanks
everybody
for
piling
in
here.
It's
gonna
be
40
minutes,
so
I'll
keep
your
legs
strong
and
we're
going
to
get
started.
So
what
I'm
going
to
be
doing
in
this
talk
is
covering
three
things.
These
are
three
very
complex
things,
often
times
requiring
a
little
bit
of
like
mind.
Explosion
recursion,
so
we're
gonna
try
to
do
our
best
to
work
our
way
through
it.
A
A
@Cd
is
a
project
we
introduced
in
2013
I'll,
be
talking
a
little
bit
about
primary
data,
store
of
kubernetes,
core
OS
Linux
rocket
and
we're
big
contributors
to
cube
entities
as
well,
and
then
we
have
Enterprise
Products
tectonic,
which
has
upstream
here
open
source
kubernetes
at
its
core
and
builds
a
bunch
of
features
around
that
that
are
necessary
for
enterprises
and
quai,
which
is
our
container
build
system
and
registry
hosted
at
quai
Thao
or
behind
the
firewall.
With
Quay
enterprise,
alright,
so
let's
get
right
into
it
with
scaling.
A
This
is
gonna,
be
the
first
topic,
I'm
essentially
gonna
cover
from
the
bottom.
Essentially,
all
the
work.
That's
happened
over
the
last
release,
or
so
from
the
very
bottom,
all
the
way
to
the
very
top
of
kubernetes.
So
let's,
let's
start
with
that,
CD
since
it'll
play
into
this
story
overall
in
several
different
places.
A
So
at
CD
is
the
primary
data
store
of
kubernetes.
It
was
introduced
in
2013
it
if
you
think
of
like
the
primary
data
store
of
something
like
WordPress
or
something
kubernetes
is
at
CD,
as
my
sequel
is
WordPress,
so
we
we
use
everything
that
kubernetes
does.
Every
time
you
write
into
the
API,
that
CD
is
persisting
to
disk
and
replicating.
A
It
allows
for
revisions
and
event
streams
so
that,
as
the
keys
change,
we're
able
to
go
back
in
time
do
things
like
consistent
views
of
the
data
five
minutes
ago
ten
minutes
ago,
and
this
is
an
important
primitive
for
kubernetes
and
some
of
the
fundamental
UX
and
design
interactions
in
kubernetes
build
upon
this
and
then.
Finally,
the
reason
that
we
replicate-
and
the
reason
the
system
is
consistent,
is
because
we
want
to
achieve
auto
leader
election
for
availability,
so
kubernetes.
A
A
Detects
failure,
auto
leader,
elects
and
allows
you
to
continue
sleeping,
while
the
computer's
figure
out
how
to
continue
working
so
for
a
quick
demo
of
just
at
CD,
just
so
everyone's
on
the
same
page
of
what
it
is
and
why
it's
built
we're
gonna
start
with
this
service
that
is
at
play
at
CDI.
Oh,
this
is
a
system
for
people
to
get
used
to
cue
brunette
or
get
you
stat
CD
and
how
it
works.
A
Now
this
is
sort
of
like
an
MMORPG
fret,
CD,
so
you'll
see
that
as
I'm
in
this
talk,
people
are
logging
in
up
in
the
corner
there,
which
means
that
generally,
what
will
happen
is
about
20
or
30
seconds.
People
start
clicking
on
the
little
circles
there
and
clicking
buttons
for
me
and
I
can
talk
through
different
failures
that
doesn't
happen.
It's
okay.
There
we
go.
So
what
Etsy
Dee
is
doing
here
is
that
we
have.
A
We
have
five
members
in
the
subsidy
cluster
and
we're
able
to
write
into
the
FTD
cluster,
submit
new
entries
in
the
sed
cluster
and
the
activity
cluster
continues
to
work
until
people
kill
too
many
members,
and
so
there's
an
in
the
symbolism.
Here.
We
have
red
means
that
the
sed
members
failed
blue
means.
The
f.2d
member
is
up
and
green
means
that
it's
the
leader,
so
I
would
I'd
kindly
like
ask
all
the
trolls
to
back
off
for
a
second
here.
A
So
I
can
have
an
operational
cluster
for
just
a
moment.
So
what
I'll
do
is
I'll
recover
three
of
these
at
CTA
members
they
come
back
up
and
now
I
have
a
working.
Cluster
@cd
is
able
to
work
with
a
simple
majority.
It
works
like
our
democracy,
which
we
will
all
see
in
action
today
and
so
I'm
able
to
submit
new
values
in
that
TD.
A
So
if
I
set
food,
a
bar
bass
or
whatever
and
I
click
Submit,
what
will
happen
is
that
this
will
replicate
to
all
the
different
things
and
the
hash
will
update,
etc,
and
you
can
see
here
that
as
I
like
click
and
kill
the
machines,
what
will
happen
is
that
leaders
automatically
get
elected?
It's
like
that.
New
HBO
show
Survivor,
alright
good
good.
So
that's
what
FTP
is
its
source,
keys
and
values.
It
replicates
those
between
machines
and
it
automatically
recovers
from
failure.
A
Any
questions
there,
hi
so
fcd
has
been
around
since
2013,
and
in
the
last
few
months
we've
introduced
a
new
version
of
that
CD
called
STD
v3.
We
did
a
little
bit
of
a
Firefox.
We
did
at
TV
0
for
at
TV
2
and
then
add
to
d3
just
skip
one
out.
Nobody
cares
so
at
2,
DB
2
was
was
designed
to
support
a
really
simple
use
case.
A
Other
parts
of
kubernetes
are
now
the
scaling
limit.
We've
pretty
much
taken
all
the
low-hanging
fruit
and
FTD
is
a
datastore
that,
as
we'll
see
here
in
a
moment,
really
shines
in
its
little
niche.
That
is
consistent,
replicated
key
value
stores.
Okay,
so
let's,
let's
take
a
look
through
history,
a
little
bit,
so
this
was
what
a
2d
compared
to
console
and
zookeeper.
Look
like
you
know
about
eight
months
ago,
so
zookeeper
is
about
at
the
top
here.
A
As
far
as
performance
being
will
handle
requests
per
second
at
CD
was
somewhere
in
the
middle
and
console
was
coming
in
third
place,
and
it
wasn't
just
about
how
many
keys
could
you
slam
in,
but
it
was
about
what
does
the
performance
look
like
when
you're
slamming
lots
of
keys
in
the
database,
and
so
a
very
common
problem
with
these
sorts
of
databases
like
that
CD?
Is
that
you
need
to
do
this
thing
where
you
have
all
this
data
streaming
in,
and
this
is
right
ahead
log.
A
So
every
time
I
change
comes
in
your
pending
this
log
and
at
some
point
the
log
grows
to
the
point
where
you
need
a
truncate.
It's
called
snapshotting
and
it
can
cause
performance
degradation
in
the
cluster,
and
so
here's
in
this
trace,
we
can
see
like
a
snapshot,
is
called
cause
Gatien
and
in
fact
later,
a
snapshot
gets
triggered
and
causes
as
a
leader
election,
because
the
Exedy
member
is
no
longer
responding,
and
this
is
common
inside
of
zoo
keeper,
as
well.
A
Snapshot
was
triggered
trigger
election
here
inside
of
a
snap
inside
of
a
zookeeper
cluster,
and
so
what
we
did
was
we
focused
on
how
to
solve
this
problem
really
nicely
with
at
CD
v3
and
you
get.
Is
this
really
tight
tolerance
band
and
this
becomes
really
important
because
you
don't
want
to
get
page
with
false
alerts
when
the
system
just
acting
normally?
Oh,
the
system
had
to
do
a
snapshot.
A
Well,
the
system
always
asses
your
snapshot,
don't
page
me
because
a
leader
election
is
happening,
and
so
we
have
this
really
consistent
band
of
latency
performance
now
inside
of
edge
TV
be
three
because
of
the
work
that
we
did
where
you
know
within
10,
20
%
we're
able
to
handle
the
same
requests
per
second
pretty
handily,
and
this
is
at
2
million
keys
at
256
bytes
per
key,
and
once
you
kind
of
zoom
out
a
little
bit,
it
looks
really
good.
We
see
these
latency
spikes.
A
The
blue
line
is
zookeeper,
which
is
like
you
know,
a
very
nice
thing
to
compare
against
pretty
battle-hardened
system
and
we
see
at
CD,
essentially
rock
steady
across
the
latency
spectrum
and
then
the
other
thing
that
we
did
was
we
wanted
to
reduce
memory.
Consumption
resident
memory,
consumption
so
at
CD
is
being
asked
by
kubernetes
to
hold
on
to
gigabytes
and
gigabytes
of
data,
and
in
order
to
do
that,
we
can't
have
this
huge
overhead
so
before.
A
We
have
this
fun
one
that
you
can
play
around
with
from
your
phone
and
troll
the
speaker,
but
also
we
have
long
long-running
ones
that
essentially
check
all
the
performance
metrics
of
that
CD.
So,
in
summary,
we
did
a
lot
of
work.
The
graphs
look
good
and
at
sea
the
kubernetes
is
in
good
shape,
riding
on
top
of
Etsy
d-iii.
So
the
story
with
that
CD,
v3
and
kubernetes
is
that
it
is
an
option
in
coop
and
I
use,
one
for
if
you
can
turn
it
on.
A
If
you
run
kubernetes
today,
on
top
of
that
to
DV
2,
you
can
deploy
out
to
DV
3.
Your
cluster
will
remain
the
same.
There
are
scripts
inside
of
the
kubernetes
repo
that
can
help
you
migrate,
your
v2
keys
and
to
be
3
keys
and
take
advantage
of
these
new
these
new
things,
but
if
you're
launching
a
new
cluster,
there's
really
no
reason
if
you're
using
one
for
or
not
you
just
turn
on
that
CV
3
plug
and
start
from
A
to
D
v3
for
all
your
new
clusters.
A
Alright,
so
we've
we've
hit
our
pretty
good
scaling
milestone
with
that
CD.
Any
questions
on
this
good.
Ok,
so
another
neat
thing
that
we've
done
in
the
last
few
days
is:
we've
introduced
this
thing
called
net
CD
operator.
The
sed
operator
is
able
to
run
kubernetes
a
run
at
CD
on
top
of
kubernetes,
so
we're
starting
to
get
really
meta
and
recursive
right
off
the
bat
here.
A
But
the
idea
is
that
we're
gonna
be
moving
to
a
world
where
we
want
kubernetes
moving
kubernetes,
which
will
be
the
next
set
of
our
slides,
and
what
the
operator
allows
us
to
do
is
really
simply
use
the
Cooper
Nettie's
API
to
manage
Etsy
itself,
so
I
have
mini
cube.
I
have
a
mini,
cube
cluster
running
on
my
laptop.
If
you're
not
familiar
with
mini
cube,
it
allows
you
run
kubernetes
release
your
laptop,
and
so,
if
I
do
coop
cuddle
get
pods.
A
What
I'll
see
here
is
that
that's
the
operator
running
and
in
a
net
city
cluster
looks
like
that's
getting
cut
off.
Is
it
getting
cut
off?
Ok,
ok,
and
so
this
Etsy
D
cluster
actually
is
using
this
concept
of
an
operator,
which
is
a
thing
that
operates
applications
for
you.
So
if
I
do
coop
cuddle
get
at
city
clusters,
what
I'll
see
is
that
there's
a
TD
cluster,
so
we've
extended
the
kubernetes
api
to
be
aware
of
SED
clusters
and
as
an
object
and
what
is
in
the
set
CD
cluster.
A
Manifest
is
really
just
two
things:
it's
what
version
of
FTD
do
I
want
running
and
the
size
of
that
CD
and
the
operator
is
actively
going
to
manage
this
cluster
for
me.
So
if
I
do
cubic,
I'll
get
pods
and
then
I'm
decide
that
you
know
I,
don't
particularly
like
this
pod
and
right
now,
I'll
delete
the
pod
and
then
what
we'll
see
happen
is
that
well
too
quickly?
Oh,
maybe
not
so
what
will
happen
is
that
HCD
operator
will
detective.
A
Correct
sorry,
so
this
is
part
of
Etsy
dv3,
so
this
utilizes,
a
lot
of
administrative
API,
is
an
density
v3
and
we
we
only
launched
EXO
DB,
3
and
above
clusters
and
these
pods,
and
so
it
took
a
second.
But
you
see
here
that
we
now
have
an
Etsy,
v3
or
Etsy
d3
pod,
that's
14
seconds
old,
so
the
entity
clusters
under
op,
active
management
and
the
operator
can
do
other
things
like
backup
at
CD
and
do
disaster
recovery
and
that
sort
of
stuff,
as
as
described
here,
there's
a
blog
post
on
it.
A
It's
all
open
source
you
can
find
it
under
github.com,
slash
core
LS,
/
@
CD
operator.
It's
a
single
deployment
to
put
it
on
your
cluster,
ok,
so
the
next
topic
is
self
hosted
self
hosted
kubernetes.
So
if
you're
not
familiar
with
the
concept
of
self
hosted,
it's
a
pretty
common
computer
science
idea.
So
in
the
case
of
a
lot
of
programming
language,
there's
their
self
hosted.
So
you
write
the
compiler
for
the
programming
language
in
the
programming
language
itself.
Your
C
compilers
not
written
in
an
assembly,
so
it's
written
in
C.
A
So
in
the
case
of
GCC,
they
take
the
GCC
source
code.
They
run
it
through
an
old
version
of
GCC
out
pops.
A
new
version
of
GCC
golang
just
became
self
posted
as
well.
So
majority
of
go
is
written
and
go
so
you
can't
we
go
source
code,
you
run
it
through
an
old
version
of
go.
You
get
a
new
version
of
go.
It
gets
a
little
bit
more
noodley
when
you
start
to
think
about
things
like
operating
systems
or
kubernetes.
So,
let's
use
an
example.
A
This
is
young
leanest,
Torvalds,
leanness
use
an
operating
system
called
MINIX
and
wrote
this
new
operating
system
called
Linux
out
pops.
Linux
neat,
I,
love,
personal
email,
notifications,
I,
don't
know
what
tab
is
doing
that,
hopefully
not
being
embarrassing
happens.
So
the
the
next
thing
that
leanness
did
you
know
lean.
It's
got
a
little
bit
older,
more
mature
and
Linux
was
actually
running.
This
actually
just
happened
in
a
few
months.
I
think
it's
funny.
A
They
use
like
a
40
year
old,
Lena's
versus
21,
and
so
Lena's
already
had
Linux,
and
so
he
took
the
Linux
source
code.
Linux
was
able
to
run
GCC
and
then
out
pops
a
new
version
of
Linux,
so
we
need
to
apply
the
same
concept
to
kubernetes.
So
today
we
have
all
these
tools
that
kind
of
deploy
kubernetes
on
our
behalf
and
then
set
kubernetes
up.
So
they
talk
to
AWS,
they
the
point
AWS.
They
do
some
configuration
management
and
then
kubernetes
api
endpoint
pops
out
at
the
end.
A
But
what
we
want
to
do
with
self-hosted
kubernetes
is
we
want
to
move
to
this
world
where,
instead
of
these
tools,
essentially
taking
care
of
this
whole
stack,
the
tool
takes
care
of
the
infrastructure,
let
the
infrastructure
tools,
focus
on
infrastructure
and
kubernetes
takes
care
of
kubernetes,
and
so
the
advantage
with
this
sort
of
platform
is
that
we
can
use,
say,
cube
kettle
and
actually
upgrade
kubernetes
and
it
works
the
exact
same
way,
whether
we're
on
AWS,
so
we're
on
Azure
or
perhaps
we're
on
bare
metal.
We
can
use
the
exact
same
API.
A
Is
the
exact
same
process
that
great
cuban
Eddie's
on
any
single
any
type
of
infrastructure
that
you
have
so
the
goals
here
with
self-hosted
is
that
we
use
the
compute,
networking
and
storage
abstractions
that
we
love
kubernetes,
for
the
reasons
that
we
use
kubernetes
in
the
first
place
is
that
it
allows
us
to
abstract
these
things.
The
infrastructure
tools
focus
on
infrastructure
and
it
provides
a
foundation
by
using
those
abstractions
to
automate
the
upgrade
of
our
clusters
and
scaling.
A
So
the
way
this
looks
in
practice
is
that
will
say:
coop
cuddle
in
coop
system
do
daemon
sets
will
see
api
server
and
proxy
running
daemon
sets
and
we'll
see,
controller
manager
and
scheduler
running
as
deployments
now.
This
has
a
lot
of
natural
advantages,
because
let's
say
that
we
want
to
scale
our
scheduler,
we
can
just
edit
appointment
of
the
scheduler
and
scale
it
so
obviously
like
the
proof
is
in
it
actually
working,
not
necessarily
looking
good
architecture
slides,
so
I've
gone
ahead
and
launched
a
self-hosted
cluster.
A
All
of
this
is
based
on
Boot
Cube,
which
is
an
incubator
project
introduced
by
core
OS,
but
we've
wrapped
really
nice
user
experiences
on
top.
So
this
is
the
essentially
installer
for
tectonic,
which
is
what
I
use,
but
underneath
it's
just
using
the
infrastructure
at
Boot,
Cube
and
out
the
other
end.
What
I
get
is
a
fully
working
kubernetes
cluster,
that's
self
hosted.
So
if
we
dive
into
the
if
we
dive
into
the
coop
system
as
described,
we
see
that
there's
the
cube
scheduler
in
here,
there's
two
pods
running.
A
We
can
you
know
if
we're
you
need
to
make
an
H
a
kubernetes.
It's
really
simple,
because
we
just
talk
to
the
kubernetes
api
and
we
say
scale
up
the
number
of
schedulers
so
that
in
case
a
couple
machines
go
away.
My
application
keeps
working
and
we
can
see
that
those
pods
get
launched
and
they
get
put
on
to
machines
that
are
appropriate
for
running
the
kubernetes
scheduler.
A
And
similarly,
you
can
come
in
and
see
the
daemon
sets
running
and
the
nodes
that
are
actually
doing
all
of
the
management
of
the
cluster
simply
have
this
label
on
them.
That
say,
master
equals
true,
which
means
it's
appropriate
for
things
like
the
scheduler
and
the
API
server
to
run
on
top
of
them,
and
so
this
is
all
using
kubernetes
api
x'.
The
great
thing
about
this
is:
if
an
API
server
goes
down
or
I
lose
a
scheduler
I
actually
can
just
debug
it.
I
can
look
through
the
events
here
inside
of
kubernetes
and
see.
A
A
That
is
a
great
question.
So
what
about
the
first
appointment
of
kubernetes?
How
do
we
bootstrap
our
way
into
it?
So
this
is
what
the
buku
tool
actually
does
and
if
you
want
to
learn
how
the
boot
queue
works,
I'll
bury
the
lede
a
little
bit.
Essentially,
what
happens
is
we
launch
a
temporary
kubernetes
as
a
single
process
inside
that
single
process?
A
Is
your
scheduling,
API
server,
@cd
everything
you
need
and
then
we
pivot
on
to
the
actual
real
kubernetes,
and
then
we
do
all
sorts
of
tricks
to
ensure
that
in
case
of
power,
outages
or
whatever,
you
don't
need
to
bring
that
temporary
Cooper
Nettie's
back
up,
and
so,
if
you
want
to
learn
about
coop
ception
Aaron
Levie
is
giving
a
really
deep
dive
on
self
hosted.
But
if
you
want
to
just
get
the
results,
you
can
just
go
on
to
be
r-cube
ball
of
AWS
and
GCP
guides
and
try
it
out.
A
The
other
thing
is
that
there's
a
lot
of
deployment
tools
inside
of
the
kubernetes
ecosystem.
Today,
there's
Cube
admin
cops,
etc.
Our
hope,
with
all
of
this
self
hosted
stuff,
is
to
show
that
it
is
a
compelling
deployment
model
show
that
it
is
stable
and
the
people
can
actually
run
real
workloads
on
top
of
it.
A
While
these
things
are
happening
and
we're
working
closely
with
the
cube
admin
team,
we've
written
up
a
design
doc
on
how
cube
admin
can
do
self
hosted,
we
are
working
with
everyone
else
in
the
Kuban
IDs
community,
including
Sigma
cluster
lifecycle,
essentially
making
this.
If
not
the
default.
At
least
one
of
the
premiere
choices
for
operating
kubernetes
inside
of
production
environments,
boot
cube,
is
really
a
tool
that
we've
been
working
on.
That
proves
out.
A
The
model
brings
all
the
things
together
and
took
he
subset,
there's
a
lot
of
work
that
needs
to
happen
inside
the
kubernetes
to
make
it
a
perfect
system
right
now,
it's
a
99%
system,
any
questions
on
self
hosting
cool,
all
right,
so
we'll
move
on
to
Federation.
This
is
a
real
whirlwind,
like
we
a
lot
of
topics
in
40
minutes
so
with
Federation.
The
big
idea
is
that
today
we
have
kubernetes.
A
This
is
what
a
single
kubernetes
cluster
looks
like
API
server
holds
on
to
the
state
of
kubernetes
it
designed
for
single
domain
failures,
so
you
want
to
put
a
single
kubernetes
cluster
inside
of
say,
a
single
data
center.
You
want
to
put
a
single
kubernetes
cluster,
say
inside
of
a
couple
of
racks
that
are
in
a
single
power
failure,
domain,
etc,
and
this
single
costs
are
going
to
handle
thousands
of
machines.
So
that's
that's
what
kubernetes
is
designed
for
single
kubernetes
cluster
is
designed
to
be
tolerant
to
individual
machine
failures.
A
Not
a
truck
hitting
your
data
center
makes
sense.
Okay,
the
idea.
What
Federation
is
that
we
kind
of
like
self-hosted,
where
we're
using
kubernetes
at
the
bottom,
the
idea,
what
Federation
is
we're
using
and
managing
multiple
kubernetes
clusters
by
putting
kubernetes
on
top?
And
so
what
Federation
looks
like
is
that
we
have
very
similar.
You
have
the
coop
cuddle
the
command
line
tool.
It's
talking
to
this
special
API
server.
That's
the
Federation
API!
It's
persisting
all
of
its
data
and
a
replicated
@cd,
that's
hopefully
running
across
whole
of
your
various
clusters
that
are
being
federated.
A
There's
a
federation
controller
manager.
That's
actually
insuring
the
workloads
end
up
on
the
machine
or
inside
of
the
clusters
that
you'd
like
to
them
that
they
end
up
into
and
then
every
single
cluster
underneath
is
just
a
normal
kubernetes
cluster.
That
actually
is
sort
of
blissfully
unaware
that
this
Federation
thing
is
actually
happening
to
it,
because
the
Federation
ap
controller
manager
is
just
another
client
scheduling,
replica
sets
or
pods
or
services
to
the
cluster.
A
So
it's
a
lot
to
digest.
Can
I
get
like
a
show
hands
if
that
all
made
sense,
great
okay,
so
that
that's
the
big
idea
with
Federation,
so
the
API
server
and
that
2d
hold
state
for
multiple
clusters
like
what
services
and
replicas
sets
you
want
running
across
your
clusters.
It
spans
multiple
failure
domains,
and
so,
ideally,
you
will
have
a
Federation
cluster,
that's
running
in
New,
York,
San,
Francisco
and
maybe
Tokyo
or
maybe
like
three
places
that
aren't
always
disaster
movies.
A
But
it's
fans,
multiple
failure,
domains
and
is
resilient
to,
like
you
know,
just
like
at
CD,
one
of
the
three
or
two
of
the
five
failure
domains
going
and
it
coordinates
between
clusters.
It's
not
actually
for
managing
the
individual
machines
in
a
cluster.
So
if
you
think
about
Federation
as
the
couplet
is
managing
processes,
the
Federation
controller
manager
is
managing
clusters.
So
it's
like
Federation's
cubelet
is
another
cluster.
A
You
if
you
want
to
go
by
analogy
okay,
so
if
you
notice
that
all
my
diagrams
are
super
slick
up
until
now,
these
I
forgot
to
update
so
you're
gonna
have
to
bear
with
me
on
this.
So
the
API
server
talks
to
an
API
server
on
the
cluster
or
actually
multiple
API
servers
and
that
API
server,
then
schedules
work
say
like
an
engine
X
process
and
then
that's
the
like
replica
set
going
through
now
with
services.
A
What
happens
is
that
the
Federation
API
it
now
has
this
replica
set
scheduled
to
the
individual
nodes,
and
then
it
creates
a
new
service
say
an
engine.
X
service
ends
up
getting
load
balance.
So
what
we're
going
to
do
is
I
have
an
actual
live
Federation
cluster
that
I've
set
up
last
night,
hopefully
so
running.
So
let's
go
ahead
and
play
around
with
that
for
a
second,
the
cluster
that
I'm
playing
with
is
set
up
using,
where
is
it
you'll
see
there?
It
is
Kelsey.
A
Hightower
is
probably
the
best
instructions
on
how
they
set
up
the
Federation
clusters.
Today
it
runs
you
through
setting
up
a
Federation
cluster
on
GCP,
along
with
all
the
load,
balancing
and
DNS,
and
everything
that
you
need
to
make
it
happen.
It
takes
about
ten
or
fifteen
minutes
to
go
through
the
entire
thing,
but
I've
done
it
for
you,
so
we
can
play
around
with
it.
A
So
the
big
things
that
happen
here
is
that
I
have
similar
to
those
diagrams.
Those
really
awesome
diagrams
that
you'll
remember
from
just
a
second
ago:
I
have
engine
acts
deployed
against
my
federated
cluster,
and
so
I
have
this
coop
kuddle
is
talking
to
this
federation
cluster,
which
is
a
subset
of
a
real
API
server.
So
if
I
try
to
run
like
git
pods,
it's
not
going
to
say
there's
any
pods
available,
because
API
server,
Federation
API
doesn't
deal
with
machines
that
deals
with
clusters,
so
it
doesn't
know
about
pods.
A
It
knows
about
replica
sets
and
services,
and
so
what
we
can
see
is
I
get
replica
sets,
and
what
we'll
see
in
here
is
that
there's
replica
sets
for
nginx
I
have
one
replica
set
that
is
replicating
only
over
the
u.s.
domains
and
they
have
another
replica
set,
that's
going
over
my
other
clusters,
and
so,
if
I
go
and
do
get
clusters,
what
I'll
see
is
that
I
have
four
clusters
in
this
Federation
shoot
sorry,
this
is
getting
cut
off.
A
This
annotation
says
I
want
this
replica
set
to
only
be
scheduled
to
us
by
us
clusters
and
I
want
there
to
be
some
policy
around
how
many
replicas
end
up
in
which
cluster
and
I
want
to
rebalance
those
in
case
anything
happens,
etc,
and
so
you're
able
to
express
really
complex
failure.
Domain
logic
inside
of
the
federation,
api
and
the
federation.
Api
takes
us
if
you
account
when
it
talks
to
the
clusters.
A
If
we
look
at
the
services
here,
so
we're
going
to
describe
the
services,
what
you'll
see
is
that
there's
a
load
balancer
that
is
federated
over
lots
of
different
IP
addresses
and
those
IP
addresses
are
the
different
IP
addresses
of
the
different
clusters
and
the
end
result.
The
big
surprise
here
is
that
we
get
a
white
engine,
XP
full-page.
A
So
this
is
a
super
highly
available
default
engine
tax
page,
it's
globally
replicated
the
DMS
is
highly
available.
It's
super
awesome,
so
if
you
wanted
to
put
like
an
yang
cat
on
here,
like
the
entire
world
would
see
permanently,
so
that's
that's
the
Federation
API
and
that's
the
Federation
API
in
practice.
Like
I,
said
it's
it's
a
work
in
progress.
This
is
what
Federation
one
plot
1.4
is
capable
of
doing.
There's
a
lot
to
be
done
left
in
the
future,
but
it
is
a
great.
A
It
is
a
great
reflection
point
and
you
can
kind
of
compare
and
contrast
it
to
self-hosted
where
self
host
is
the
bottom
kubernetes
on
kubernetes
cluster
Federation
as
a
top
kubernetes
on
kubernetes,
it's
a
pretty
neat
neat
model
and
really
shows
off
the
power
of
kubernetes
as
a
platform
for
not
just
handling
our
stateless
web
applications
or
our
twelve
factor
apps.
But
of
handling
these
complex,
these
complex
deployments
that
a
lot
of
us
find
ourselves
managing.
Yes,.
A
Absolutely
great
question:
so
how
does
high
availability
of
Federation
work
the
going
back
to
these
great
diagrams?
So
what
happens?
I
have
deployed
the
federation
api
in
a
single
cluster.
Obviously,
that's
not
what
you
want.
You
want
to
add
cv
replicated
over
lots
of
things,
but
it's
really
hard
to
set
up
so
I
didn't
because
I'm
lazy.
The
way
it
should
work
is
that
you
essentially
run
the
Federation
API
server
over
all
of
your
clusters,
along
with
that
replicated
at
CD,
and
these
are
replicating
essentially
globally,
and
so
these
are
put
by
and
load
balancers.
A
Those
load
balancers
end
up
in
some
sort
of
globally.
It
depends
on
how
you
want
to
set
this
up
if
your
cloud
provider
can
handle
it
or
your
isp
or
whoever,
or
maybe
you
put
DNS
records
with
really
low
TTLs,
but
you
put
those
load
balancers
inside
of
a
DNS
record,
say
in
the
naive
case
and
then,
if
any
one
of
those
data
centers
goes
down,
you
either
trigger
automatically
through
like
global
monitoring.
A
B
A
Yeah
for
sure
like,
if
you
overload
your
clusters,
like
that's
gonna,
cause
problems.
The
advantages
that
we
have
is
that
containers
allow
us
to
do
better
resource
isolation,
where
ideally
you're
able
to,
through
the
cubelet
and
through
the
kubernetes
isolation
of
namespaces,
we're
able
to
actually
ensure
that
we
don't
over
schedule
or
over
provision,
but
assuredly
just
like
a
for
loop
or
parallel
SH
like
you,
can
definitely
get
yourself
in
a
massive
bind
with
these
sorts
of
systems.
It
just
comes
down
to
monitoring
and
good
good
practices.
Yeah.
B
A
A
See
so
the
advantage
that
something
like
Federation
has
is
that
it's
kind
of
like
pushing
down
from
the
top
and
then
recovering
events
back
through,
and
so
you
absolutely
can
like
end
up
it
in
places
where
bugs
and
stuff
will
take
out
the
whole
system,
but
is
very
much
a
like
tree
based
system
where
the
root
is
pushing
down.
There's
not
really
a
lot
of
like
RPC
happening
between
the
two
systems,
actively
to
like
do
active
help
on
attorney
that
sort
of
stuff.
So
it's
I'm
definitely
like
dodging,
but
it
is
like.
A
Yes,
so
I'm
super
punting
on
load-balancing
by
using
the
Google
cloud
load,
balancers,
so
they're.
Actually,
the
Federation
API
is
like
setting
up
all
the
DNS
records
under
GE,
FF
org,
and
so
it's
setting
up
all
the
Federated
DNS
records
and
then
creating
load
balancers
and
managing
all
that.
For
me,
the
Federation
API
is
doing
that
on
my
behalf,
but
I'm
kind
of
like
punting
on
that
Hart.
A
A
A
Yeah,
so
right
now
only
replica
sets
are
supported.
You
can't
really
do
like
deployments,
and
so,
when
you
like,
update
the
replica
set,
you
yeah
it'll,
essentially
like
the
Federation,
API
and
controller
manager,
are
pushing
like
you
need
to
be
running.
These
replica
sets.
Are
these
these
pods
pushing
it
down
to
the
cluster?
A
A
Correct
okay,
so
the
clusters
are
acting
up
independently
yeah,
so
the
Federation
API
supports
ingress
services,
namespaces
replicas
stats
secrets
and
DNS,
which,
as
I
mentioned
as
cloud
provider
specific
today.
So
the
next
steps
with
Federation
there's
to
check
out
the
Federation
setup
guides
at
kübra
I/o.
They
kind
of
give
an
overview
of
how
the
whole
thing
works.
Kelsie's
guides,
like
I
mentioned,
are
easier
than
the
official
guides
right
now,
so
you
may
want
to
check
those
out.
Sig
Federation
meets
regularly.
A
There's
a
Clinton
leads
sig
Federation
he's
in
the
back
of
the
room,
they're
wearing
a
bright
blue
kubernetes
shirt.
So
talk
to
him.
If
you're
interested
in
this
and
it's
it's
a
interesting
project
and
rapidly
developing.
So
what
all
these
disparate
things
have
in
common?
Why
did
I
throw
them
all
into
one
talk?
One
is
that
I
didn't
think
ahead
to?
Is
that
there's
actually
like
shared
vision
here
so
with
these
operators
operating
at
CD,
we
can
have
at
CD
going
across
and
actually
handling
this
complexity
of.
A
How
do
I
get
at
CD
replicated
across
these
different
AZ's
self-hosted?
You
can
imagine
starting
to
manage
kubernetes
itself
and
Federation
managing
lots
of
kubernetes
clusters,
so
these
abstractions
are
very
similar,
and
one
thing
I
just
want
to
continue
to
point
out
is
this-
is
all
leveraging
tested
designs.
We
trust
kubernetes
and
the
federation.
Api's
are
very,
very
similar,
abstractions
and
very,
very
similar
designs.
So
my
goals
from
here
I
want
to
make
a
self-hosted
the
default
for
communities
clusters,
or
perhaps
at
least
the
the
preferred
deployment
model.
A
I
want
to
make
it
easy
to
operate
at
2d
for
single
and
federated
cases.
That's
why
we've
introduced
the
sed
operator
work
that
we've
done
and
when
I
bring
these
Federation
technologies
to
market,
because
I
think
over
the
entire
time
that
I've
been
involved
in
the
cloud
infrastructure
market
over
the
last
eight
or
so
years.
All
of
it
I've
ever
heard
about
is
hybrid
cloud
and
gosh
darn
it.
We
want
to
bring
hybrid
cloud
actually
to
market
I
think
this
is
the
probably
the
best
way
in
the
best
shot
that
we
have
doing
that.
A
If
you
want
to
learn
more
about
this
stuff
for
what
we
do
at
coral
s,
or
you
happen
to
be
in
New
York
in
December,
we
have
a
conference
December,
12th
and
13th
a
bunch
of
companies
getting
together
talking
about
how
they're
using
kubernetes
how
this
whole
cloud
native
thing
is
working
inside
of
their
organizations
and
with
that
I'd
like
thank
you
for
your
attention
and
I'll
take
a
few
more
questions.
We
have
about
three
minutes,
and
that
is
that.
Thank
you.
A
A
That
you
discussed
yeah,
so
this
is
a
question
that
operators
grow
us
if
you
go
read
the
blog
there's
an
FAQ
at
the
bottom,
about
that.
The
big
thing
is
that
helm
charts
are
about
packaging
up
a
set
of
essentially
kubernetes
resources
as
like
a
single
object,
whereas
an
operator
is
actively
managing
a
complex
piece
of
software
and
doing
on
behalf
of
the
user,
like
a
lot
of
the
tasks
that
they
would
have
been
thinking
about.
A
Right
so
the
question
was
our
operators
knew
primitive
of
the
kubernetes
api.
It's
really
a
pattern
of
that
contains
two
things.
The
kubernetes
has
always
been
able
to
you.
It's
always
been
able
to
have
external
controllers,
so
these
pieces
of
software
that
act
on
behalf
or
act
on
the
kubernetes
api
running
on
the
kubernetes
api
and
the
idea
of
a
third-party,
odd
object
or
third-party
resource,
and
so
combining
these
two
things.
There's
active
management
component
and
this
idea
of
a
third-party
resource
is
what
makes
an
operator
an
operator.
C
Is
the
Federation
API
server
developing
out
of
the
existing
API
server,
or
is
it
a
something
else
I
mean
built
from
scratch?
Yeah
second
question
is
the
HDD
cluster
operators,
or
otherwise
should
I
treat
them
as
something
part
of
infra
kubernetes
or
should
I
treat
them
as
HCD
as
a
service
and
use
it
for
applications
and
kubernetes
together?
What's
the
recommended
way,
yeah.
A
So
good
questions
on
Federation
API.
It
is
leveraging
essentially
all
the
same
as
factions.
The
API
server
has,
but
it's
a
subset
and
operates
differently
than
normal
API,
for
example,
like
git
pods,
isn't
a
resource,
so
a
lot
of
abstractions
that
are
being
leveraged
on
@cd.
We
really
so
FTD
is
a
very
performance,
sensitive
application.
A
It
is
doing
its
best
to
essentially
do
as
by
analogy
like
raid
between
lots
of
machines
only
for
this
important
configuration
data,
and
so
our
recommendation
is
that
you
use
a
single
at
CD
for
kubernetes
and
things
like
operators
make
it
easy
for
you
to
launch
traditional
pet
CDs
for
your
individual
applications
in
a
lot
of
cases.
Some
of
the
really
critical
applications,
like
your
network
systems,
like
flannel
canal
project
calico,
weave
et
cetera,
are
moving
to
running
on
top
of
the
kubernetes
api
themselves
instead
of
talking
about
cv
directly.
A
But
there
are
a
lot
of
applications
like
vault
or
traffic
or
others
they
want,
or
sometimes
you
utilize
at
CD
directly
so
for
those
applications
that
totally
makes
sense
to
manage
your
own
or
at
least
deploy
a
net
CD
for
those
yeah.
Sorry,
you
asked
the
question
really
early,
so
I
was
rupiah
in
the
Federated
set
up.
It
could
be
a
multi
cloud
between
different
public
clouds
yeah
exactly
so
that's
the
entire
goal.
So
the
question
was
about
multi
Club.
A
The
entire
goal
is
that
you're
able
to
initially
it's
probably
easiest
to
be
like
I'm
gonna,
run
in
multiple
AWS
regions,
but
you
could
imagine
essentially
being
able
to
start
to
play
the
games
where
I
run.
You
know
on
Azure
I
run
on
AWS
and
I
load
balance
between
those,
maybe
I.
Don't
trust
one
of
the
providers
or
I'm
gonna
start
to
play
games
where
I
noticed
pricing,
differences
and
I've
run
my
job,
spinning
where
the
pricing
is
good.
So
that's
absolutely
the
goal
and
the
plan
yeah
I'll
take
a
little
overtime.
D
A
A
Absolutely
yeah,
and
so
in
that
case,
like
it's,
not
gonna,
be
the
world's
fastest
Federation
API,
but
it
will
get
the
it'll
replicate.
The
data
correctly
right
and
you
want
all
you're
thinking
about-
is
tuning
that
election
time
out,
so
that
so
that
it
is
the
correct
latency
that
you
expect
over
your
transatlantic
link
or
whatever
and.
C
A
Is
exactly
if
you,
you
read
some
of
the
material
from
the
Google
site,
reliability,
engineering,
books?
They
have
this
idea
of
cluster
chubby
Chubby's
like
the
equivalent
of
Etsy
D
inside
of
Google.
They
have
you
have
clusters
chubby
and
global
chubby,
and
so
what
we're
creating
with
the
Federation
at
CG
is
essentially
the
global
chubby
yeah
terrible
naming
all
right.
I've
I
have
Google
or
I've
more
stickers
up
here.
I'll
take
a
few
more
questions
and
they
hang
out.
But
thank
you
so
much
for
your
attention.