►
Description
Join Scott Weiss, Architect at the Office of the CTO, on our next Live Hoot Episode on May 4th. Scott will dive into Building Multi-Cluster Operators for Kubernetes.
In the Kubernetes ecosystem, more users than ever are adopting multi-cluster architectures, where multiple Kubernetes clusters can be managed from a single pane of glass. Existing projects such as Kubefed, Istio, GlooMesh, and Admiral already provide the capability to manage Kubernetes resources across multiple clusters. This presentation will explore possible architectures and various ecosystem implementations of Kubernetes Operators designed to manage multiple clusters.
About us https://www.solo.io
Questions? https://slack.solo.io
Code Samples: https://github.com/solo-io/hoot
Suggest a topic to cover here: https://github.com/solo-io/hoot/issues/new?title=episode+suggestion:
A
A
Okay,
there,
we
go
all
right.
Okay,
great
hey
everybody!
Welcome
to
another
edition
of
the
hoot,
I'm
scott
weiss
architect
at
solo
I
o
I'll,
be
presenting
to
today
about
an
area.
We've
spent
a
lot
of
time
working
on
at
solo,
which
is
building
multi-cluster
operators,
I'm
so
hearing
that
go
of
myself
all
right.
A
So,
yes,
so
building
multi-cluster
operators
for
kubernetes
and
we'll
explain
what
all
that
means,
but
essentially
what
this
is
going
to
cover
is
when
we
build
operators.
You
know
we
write
our
own
operators
to
automate
processes
in
kubernetes,
how
we
can
build
those
operators
to
manage
multiple
clusters,
not
just
a
single
cluster
and
we'll
get
into
some
of
the
solutions
and
architectures
and
motivations
for
that.
A
A
You
know,
basically,
multi-cluster
is
becoming
more
and
more
important
part
of
the
kubernetes
story,
seeing
how
we
scale
out
our
kubernetes
adoption.
Also
kubernetes
isn't
just
being
deployed
multiple
clusters
in
a
single
provider,
a
single
infrastructure,
we're
seeing
multiple
infrastructures
in
hybrid
cloud
and
multi-cloud
deployments
of
clusters,
so
just
to
understand
why
users
would
have
multiple
clusters,
the
different
use
cases
basically
fall
into
two
categories.
A
You
can
have
replicas
of
an
existing
cluster
that
may
be
in
different
availability
zones.
It
may
be
on
different
clouds
or
in
different
and
on
prime
and
off-prem,
but
we
have
essentially
what
this
this
cluster
is
running
itself.
It
tends
to
be
the
same.
The
idea
is
that
we're
basically
running
replicas,
just
like
we
can
have
multiple
replicas
of
a
deployment.
A
We
can
also
have
multiple
replicas
of
an
of
a
cluster
deployed
to
different
places.
This
allows
us
to,
for
example,
replicate
our
food
service
here
in
multiple
regions,
so
that
an
api
gateway
or
load
balancer
could
route
between
them,
for
example
based
on
locality,
based
on
where
a
user
is
sending
their
traffic
from
or
based
on,
what's
available.
So
if
one
cluster
is
unhealthy,
we
can
route
traffic
for
the
target
service
to
a
different
cluster.
A
So
this
is
the
homogeneous
approach
and
and
use
cases
for
homogeneous
clusters.
Then
you
have
heterogeneous
clusters
and
the
use
case
for
these
is
when
you
actually
want
to
have
a
separation
of
concerns
that
build
the
application
within
the
kubernetes
clusters
themselves,
and
this
you
tend
to
see
with
large
organizations
where
they
have
separate
platform
team
and
application
teams.
A
So
often
application
teams
will
be
given
their
own
clusters,
but
the
resources
that
those
clusters
are
privy
to
may
be
restricted
and
they
may
have
to
go
through
an
additional
cluster
in
order
to
access
apis
or
more.
This
is
just
one
example
of
how
multiple
clusters
can
be
used
to
separate
concerns.
There
may
be
organizational
change
requirements
that
necessitate
different
organizations
have
their
own
kubernetes
clusters.
There
may
be
tenancy
concerns
where
each
tenant
is
given
its
own
kubernetes
cluster.
A
There
are
security
reasons
why
different
services
may
be
isolated
into
different
clusters,
but
essentially
these
are
the
two
classifications
for
multi-cluster
and
we
see
it
solo
working
with
customers,
both
use
cases
being
adopted
depending
on
the
organization.
Maybe
they
may
be
using
one
or
both
types
of
multi-cluster
scenarios,
so
setting
running
a
multi-cluster
setup
like
so
whether,
regardless
of
the
case,
requires
us
to
solve
two
problems
that
come
up.
One
is
about
orchestration,
which
is
how
do
we
actually
orchestrate
the
resources
across
multiple
clusters?
A
But
you
know
the
naive
approach
would
be
to
simply
apply
manage
configuration
for
each
individual
cluster
individually
and
apply
it
individually,
and
then
you
start
to
get
to
almost
more
of
an
imperative
flow
where
we
have
to
iterate
through
all
of
our
configurations
and
apply
them
individually
to
each
cluster,
which
many
customers
are
actually
doing
today.
Many
users
are
doing
today
themselves
by
hand
in
their
ci
cd
pipelines,
for
example,
and
that's
a
sort
of
naive
implementation
or
a
homegrown
implementation
of
the
simplest
approach
to
configuring
multiple
clusters.
A
But
that's
that's
for
the
orchestration
or
the
configuration
and
then
there's
the
networking
and
networking
essentially
says
once
we've
actually
configured
these
clusters
or
once
we've
figured
out
how
to
configure
these
clusters.
How
do
we
then
connect
the
services
running
within
them
to
each
other
and
allow
cross-cluster
communication?
A
This
is
especially
important
in
the
heterogeneous
syst
architecture,
where
services
may
depend
on
each
other
services
that
that
depend
on
on
different
service.
In
a
different
cluster,
there
also
may
be
a
case
where
service
is
talking
to
a
local
version
of
another
service
that
it
depends
on,
but
that
service
is
replicated
in
the
local
cluster
and
a
remote
cluster,
and
if
the
local
cluster
one
becomes
unhealthy,
we
may
want
to
fail
traffic
over
to
the
remote
cluster,
which
would
also
require
this
interconnectivity
between
clusters.
A
So
this
talk
is
not
going
to
go
into
the
networking
side
of
things.
We
have
another
hoot
covering
that
and
if
there's
more
info
there's
more
interest,
it's
really
a
very
dense
topic
and
we
could
spend.
We
could
definitely
if
there's
interest,
let
us
know,
send
us
a
hoot
and
we'll
you
know.
We
can
definitely
cue
up
some
more
talks
to
do
more
deep
dive
and
how
how
to
accomplish
the
networking
piece,
but
just
from
a
high
level.
A
Orchestration
today
can
be
done
using
git
ops.
Like
I
mentioned,
you
can
just
integrate
your
git
apps
process
in
order
to
essentially
individually
configure
each
cluster,
but
do
so
in
a
way
that's
easily
managed
from
a
single
git
ops.
Workflow
like
this
can
be
done
with
argo.
Today,
argo
has
custom
management.
I
believe
flux
does
as
well,
which
are
two
cicd
projects
in
the
kubernetes
ecosystem.
A
This
is
how
some
projects
like
admiralty
and
tensile
kube
are
doing
it
where
you
basically
have
one
management
cluster,
and
then
you
you,
take
your
sub
clusters
or
your
managed
clusters,
and
you
consolidate
them
into
a
virtual
kubelet,
and
you
have
a
virtual
kubelet
added
to
your
main
kubernetes
cluster
as
a
node
and
it'll.
Then
it
can
then
schedule
work
like
pods
and
such
onto
those
remote
clusters.
A
We
give
it.
We
install
an
agent
on
each
of
the
remote
clusters
and
then
that
agent
does
all
of
its
local
kubernetes
operations
and
they
cluster
together,
using
what
we
call
the
relay
api,
which
allows
a
central
management
plane
to
relay
instructions
to
an
agent
running
on
the
local
machine.
So
it
can
read
and
write
resources
to
look
to
remote
kubernetes
without
needing
to
directly
access
the
cube
api
of
those
clusters.
A
A
So,
for
example,
you
may
be
able
to
set
up
a
vpc
for
your
all
the
vms
on
which
kubernetes
is
deployed
and
then
deploy
your
pause
on
a
subnet.
That's
addressable
from
both
you
know,
let's
say
both
of
your
vms
and
and
essentially
set
up
flat
networking
for
all
the
pods.
This
really
has
to
do
with
the
way
the
clusters
are
set
up
and
depends
on
your
cloud
provider
and
what
they
support.
A
But
this
would
allow
pods
to
talk
to
each
other's
ips
directly.
It
doesn't
necessarily
solve
the
problem
of
dns
names
and
how
services
actually
will
get
routed
can
do
service
discovery
essentially
or
can
talk
to
those
remote
ips
on
a
dns
name,
so
service
mesh
provides
some
functionality
there.
Service
mesh
also
provides
us
the
ability
to
reroute
traffic
through
proxies
that
intercept,
calls
that
are
cross
cluster
and
then
route
them
through
something
like
an
ingress
load
balancer
on
a
remote
cluster.
A
Then
you
also
have
various
cni's
such
as
psyllium
that
you
can
install
in
kubernetes
and
those
can
be
configured
to
create
a
virtual
virtualized
flat
network
across
network
segmented
networks,
and
then
you
can
also
just
use
your
own
custom
setup.
You
can
install
an
nginx
or
glue
load
balancer
inside
your
cluster
and
just
configure
your
services
that
need
to
talk
across
cluster
to
go
through
the
ingress
of
the
remote
clusters
where
they
want
to
talk.
A
A
There's
some
approval
process,
maybe
a
puller
request
gets
merged
and
this
initiates
the
cicd
pipeline.
The
cicd
pipeline
takes
all
of
the
configuration
that's
stored
in
the
git
repo.
There
may
be
configuration
assigned
to
each
individual
cluster
or
there
may
be
templates
of
configuration
that
are
applied
unanimously,
with
only
those
templated
out
changes.
As
they
get
applied
to
each
cluster,
but
the
configurations
get
applied,
any
pods
that
need
to
be
scheduled
services
need
to
be
created,
ingresses
service,
mesh
configuration,
etc.
A
So
this
means
that
a
static
configuration
will
be
applied
to
the
clusters
on
a
git
repo
update,
but
the
clusters
themselves
being
updated
or
state
changing
inside
the
cluster.
For
example,
a
node
going
down
wanting
to
change
the
way
or
cluster
going
down
and
wanting
to
reschedule
the
workloads
onto
a
different
cluster
is
not
something
that
is
handled.
Natively
by
this
system
would
need
more
manual
user
intervention
in
order
to
support
a
more
dynamic,
more
responsive
approach.
A
So
this
is
how
some
projects
out
there
like
admiralty
and
liko,
are
working
and
it's
primarily
useful
for
scheduling,
work,
scheduling,
pods
and
other
things
that
actually
get
scheduled
to
kubernetes
notes
where
other
types
of
configuration
which
an
admin
may
want
to
apply
to
a
specific
cluster.
For
example,
some
kind
of
networking
or
ingress
are
not
necessarily
going
to
be
represented
by
this
approach.
A
So
this
is
not
going
to
support.
For
example,
a
cluster
admin
wanted
to
configure
a
specific
cluster
being
managed
again,
it
depends
on
how
the
virtual
node
is
implemented,
but
in
the
existing
solutions
out
there,
we
mostly
see
this
being
used
to
manage
scheduling,
pods
or
other
types
of
workloads
like
vm
workload.
A
So
let's
take
a
look
at
the
multi-cluster
controller,
because
I
think
this
is
the
most
conformant
with
what
we
expect
from
a
from
in
a
kubernetes
ecosystem.
The
way
we
like
to
work
with
things
is
to
have
a
single
declarative
configuration
file
that
can
be
then
be
applied
to
ensure
a
desired
state
in
my
infrastructure
and
as
as
whether
or
not
that's
a
single
kubernetes
cluster
or
multi.
A
You
know
privileged
that
we
don't
want
to
give
to
the
controller.
So
what
we'll
often
do
is
we'll
provision
a
service
account
on
the
workload
cluster.
Then
we
take
that
service
account
and
we
bundle
the
service
account
token
into
a
new
coop
config,
and
we
ship
that
coupe
config
back
to
the
controller.
Where
then
the
controller
can
access
it,
and
now
that
coupe
config
is
constrained
by
the
rbac
that
we've
applied
to
the
service
account
that
lives
on
the
remote
cluster.
A
It
also
allows
those
controllers
to
implement
new
apis
which
address
multi-cluster
use
cases.
For
example,
a
cluster
admin
can
provision
a
crd
which
requests
a
service
mesh
or
an
orchestration
component
to
flatten
the
network
or
create
a
bridge
network
between
a
set
of
workload
clusters
to
enable
cross-cluster
communication,
and
then
the
controller
can
go
and
individually
assign
the
service
mesh
config
to
a
mesh
running
on
each
cluster,
which
can
then
bridge
them
together.
A
And
finally,
just
to
review
the
the
updated
approach
to
this
model
to
the
multi-cluster
controller
that
we've
begun
implementing
here
at
solo,
I
o,
rather
than
having
the
controller
directly
access
the
remote
clusters
via
a
coupe
config.
Instead,
we
join
an
agent
that
runs
on
the
remote
cluster
back
to
the
controller,
using
a
shared
root
of
trust.
So
in
this
model
we
don't
need
to
provision
a
a
new
credential
on
the
workload
cluster
and
then
ship
it
back
to
the
controller.
A
A
It's
called
the
kubefed,
which
is
one
of
the
popular
implementations
of
a
multi-cluster
controller.
That's
designed
for
sort
of
generic
use
case,
which
is
like
I
have
some
yaml.
I
want
to
distribute
across
clusters,
you
use
the
coop,
fed
ctl
cli
command
in
order
to
join
the
cluster
and
it
orchestrates
that
flow.
A
But
one
of
the
advantages
of
this
agent
workflow
is
that
the
agent
just
comes
online.
It's
given
that
client
certificate
to
talk
to
the
controller
and
establish
its
identity,
and
it
simply
just
connects
and
then
begins
managing
its
local
api
server.
So
there's
no
need
for
an
exchange
of
credentials,
there's
no
need
to
provision
something
wait
and
then
manually
ship
that
into
a
different
cluster.
A
That
would
be
one
of
the
advantages
here.
Another
advantage
is
that
the
agent
can
do
pre-processing
of
resources
and
data
so
that
if
the
controller
wants
to
watch
something
or
do
a
read
on
the
remote
cluster,
it
can
request
from
the
agent
only
the
subset
of
the
minimal
subset
of
resources
that
it
needs
and
the
agent
can
parse
down
that
data
into
something
more
lightweight.
A
A
A
A
No
questions,
okay,
so
as
far
as
a
demo,
it's
I
really
wanted
to
leave
it
up
to
the
audience
to
decide
what
you'd
like
to
see,
because
there
are
a
few
options.
One
thing
I
could
do
for
you
is
demo
coop
fed
and
actually
show
what
it
looks
like
to
put
some
config
on
one
cluster
and
move
it
to
another.
A
Another
thing
is
this:
this
talk
is
about
building
controllers,
so
I
can
actually
show
you
what
it
looks
like
to
write
a
controller.
We
could
do
it
in
go
using
the
controller
runtime
library,
which
is
you
know
what
we
use
at
solo.
I
o
and
and
show
you
what
a
multi-cluster
controller
would
actually
look
like
in
terms
of
the
code,
but
I'd
like
to
gauge
from
the
audience.
If
there's
any
what
what
your
interest
would
be
and
then
we
can,
we
can
take
it
from
there.
A
The
the
type
of
thing
that
you're
looking
for
here,
we
could
also
there's
a
some
more
slides.
I
have
here
on
the
security
constraints
and
why
we're
going
with
this
new
approach,
although
I
still,
I
think
that
that's
it's
probably
better
suited
for
its
own
talk,
so
we
can
get
into
the
deeper
technical
details.
A
A
Okay,
okay,
very
cool,
very
cool,
more
interested
in
writing
the
multi-plus
controller-
that's
that
was
what
I
figured
would
be
more
interesting
because
trying
to
leverage
tools
from
the
community
from
the
ecosystem
is
you
know
it's
it's
probably
the
first
avenue
that
that's
worth
investigating
or
considering,
depending
on
the
use
case.
But
in
this
scenario
we
want
to
write
a
generic
controller.
That's
going
to
oh,
my
obs
is
freaking
out
a
little
bit
here,
yeah.
So
let's
take
a
look
at
what
writing
a
multi-cluster
controller
looks
like
so.
A
A
A
A
All
right
first
thing
that
we
want
to
do
when
we
create
our
controller.
Is
we
want
to
connect
to
the
local
kubernetes
cluster?
So
we
need
a
config
and
this
controller
runtime
library
offers
that
to
us-
and
let
me
just
show
you
it's
this
controller
runtime
here
you
just
do
go
get
six
kate,
so
io
controller,
runtime
and
that'll
that'll
get
it
for
you
in
your
local
go
project.
A
Then
you
get
a
config.
This
is
called
a
rest
config
this.
This
is
basically
your
kubernetes
config
it'll.
It
can
get
loaded
into
and
in
the
if
this
is
running
on
your
local
machine.
It'll
use
your
kubernetes
config,
and
if
you,
if
it's
running
inside
of
a
pod,
it's
going
to
automatically
load
it
from
the
the
service
account
credentials
that
the
pod
has
access
to.
So
you
can
see
they
have
this
in
cluster
config
see
it
reads
it
from
this
file
inside
of
the
pod.
A
So
anyway,
this
rest
config
is
going
to
become
important
in
a
second.
But
right
now,
when
we
run
this,
this
means
we're
getting
it
either
for
the
cluster
that
we're
living
in
or
the
cluster
that
our
coop
config
is
pointing
at
if
we're
running
on
on
our
local
machine-
and
this
is
what
we're
going
to
treat
as
our
management
cluster,
so
we're
going
to
take
our
there's.
This
thing
called
a
manager.
A
A
A
I
mean
that's,
basically
it
that's
really
all
we
really
need
here,
there's
there's
more
to
it
than
that,
but
or
there's
more
that
the
manager
exposes,
but
essentially
I
get
the
client
we
get
the
cash
between
those
two
things.
You
have
pretty
much
everything
you
need
to
build
a
controller
for
most
use
cases,
but
anyway
we
create
a
manager.
Now
this
essentially
allows
us
to
start
watches
on
things
in
the
cluster,
as
well
as
to
write
things
to
the
cluster.
A
A
A
A
A
A
All
right
now
we
have
a
watch
here.
It's
going
to
take
a
source,
so
you
can
make
your
own
sources.
Source
is
basically
a
source
for
events.
It
tells
the
controller
runtime
how
to
feed
events
into
your
reconciler
that'll
call
the
reconcile
function,
which
we
actually
haven't
defined.
Yet
it's
going
to
go
right
in
here,
but
let's,
let's
figure
out
what
we're
reconciling
first
before
we
figure
out
how
we're
reconciling
so
I'm
going
to
put
in
this
here
source.kind-
and
this
is
a
way
of.
A
A
A
A
And
that's
essentially,
what
is
this
says,
watch
config
maps
and
then
generate
a
request.
That
is
the
name
namespace.
You
can
see
it
right
here.
It
adds
to
the
queue
every
time,
there's
an
event
like
a
create
event,
an
update
event
or
delete
event
and
to
be
clear
this
this
refers
to
the
event
is:
what
happens
when
the
controller's
cache
that's
running
locally,
synchronizes,
an
event
from
the
server,
so
the
server
says
I
have
four
config
maps
and
then
the
client,
the
the
the
controller
here,
says.
A
Okay,
I
have
four
create
events
because
an
event
an
object
just
got
created
inside
of
my
cache
and
every
time
your
controller
gets
restarted.
It'll
get
create
events
for
everything
that
every
time
its
cache
becomes
populated.
So
just
to
be
clear,
these
events
track
what
happens
in
the
client-side
cache.
That's
important,
it's
not
what's
tracking
on
the
server
and
there's
no
guarantee
it's
guaranteed
to
be
eventually
consistent.
A
A
A
Let's
see,
replies
no
replies,
all
right,
I'm
going
to
move
along
then
so
we've
set
up
a
watch
now,
what's
going
to
happen
when
our
cache
picks
up
a
config
map.
Well,
we
haven't
decided
that
yet
because
this
reconciler
here
we
haven't
defined
the
reconciler,
defines
the
function.
That'll
be
called
slowing
down.
Again,
I
think
it's
it's
the
new
macbooks
they
just
they
don't
have
the
processing
power
to
handle
something
like
obs
and
go
land.
At
the
same
time,
it's
I
think
we're
coming
up
on
bottleneck.
A
A
A
A
This
reconciler
here
is:
what's
going
to
be
called
when
we
in
queue
events,
it's
going
to
receive
these
or
these
requests,
so
we
have
to
define
the
reconciler.
Let's
see,
reconciles
reconcile
reconciles
an
object.
What's
the
interface,
the
interface
is
just
a
single
function
reconcile
and
it
gets.
This
request
object.
A
A
A
A
A
A
A
A
All
right
now
you
see
we
are
caches
synchronizing
here
and
we're
receiving.
First,
we
get
every
single
config
map
that
exists
in
the
cluster
and
then
you'll
see
some
of
them
are
getting
changed.
They're,
probably
you
see,
these
are
locks
a
lot
of
the
time.
Different
components
will
use
config
map
actually
as
a
as
a
locking
mechanism,
so
they
update
the
annotations
and,
let's,
let's
take
a
look,
we
can
actually,
let's
take
a
look
at
this
ingress,
gce
lock,
for
example.
So
if
I
get
in
the
cube
system,
namespace.