►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Kubernetes Multi-Cluster Operations without Federation - Rob Szumski, CoreOS (Intermediate Skill Level)
Federation is typically thought of as the only way to do multi-cluster operations, but that's not the case. I explore more secure and scalable methods for connecting clusters together. This is key to hybrid scenarios where you want two or more clusters set up in a consistent way, and then deploy an application in all regions or on all clouds. I will dive deep into sharing RBAC roles and resource limits, plus setting up the same namespaces with important default config like Pod Security Policies.
About Rob
Rob Szumski is the Tectonic product manager at CoreOS, a provider of infrastructure software to securely run application containers anywhere. Focused on product design from open source to commercial offerings, Rob has deep experience in UX, web interfaces, and CLI. Rob likes to experiment with the technology he works on to understand it from both the developer's and the end user's side, which informs his design decisions. Prior to CoreOS, Rob was a product designer for developers in the cloud division at Rackspace.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
The
first
thing
that
we
learned
when
we
were
thinking
about
putting
at
Federation
or
multi
cluster
like
features
end
at
tecktonik,
was
that
everybody
is
running
multiple
clusters
and
there's
a
variety
of
reasons
for
this
you've
got
your
dev
test,
prod
type
setup
that
completely
makes
sense.
You've
got
folks
that
are
running
in
every
region
of
Amazon,
for
example.
This
is
very
common.
A
If
you
have,
you
know
a
points
of
presence,
type
model
for
your
software
and
you
want
to
be
in
every
region,
pick
your
favorite
cloud
and
then
you
don't
want
to
pick
your
favorite
cloud.
You
want
to
be
on
all
the
clouds
or
you
want
to
be
on
prim
data
centers
bursting
to
the
cloud
ton
of
use
cases
here
and
then,
of
course,
what
that
ends
up
looking
like
is
this.
This
is
impossible
to
manage.
You
have
to
track
all
these
clusters.
You
have
to
keep
them
up-to-date.
A
You
have
to
get
access
controls
straight
across
all
of
them,
it's
very,
very
confusing.
So
let's
explore
that
and
does
anybody
have
a
guess
what
the
average
number
of
clusters
that
one
of
our
enterprise
customers
has
all
right?
What
you're
running
today
is
it
5?
Is
it
10?
Is
it
15
some
people,
you
know,
as
we
saw
in
the
keynote
earlier
run,
a
ludicrous
amount
of
them.
We
saw
a
less
than
ludicrous
5
to
10
was
the
average
that
folks
were
running,
and
so
what
this
means
is
the
hybrid
story
is
real.
A
This
is
needed,
and
this
is
why
everybody
is
interested
in
Federation,
while
you're
all
here
today,
and
so,
let's
jump
into
what
Federation
looks
like
today.
Just
a
level
set
before
we
look
at
the
future
and
Federation
is
really
you've
got
these
two
cluster's.
It's
a
set
of
nodes
and
you've
got
some
API
servers
and
you
want
to
connect
some
workloads
between
them.
So
what
do
you
do?
Oh,
let's
just
take
the
same
API
server
and
just
call
it
the
Federation
API
server
and
throw
it
on
top.
A
So,
instead
of
scheduling
out
your
API
server,
two
nodes,
we've
got
a
new
federated
API
that
just
schedules
out
to
other
API
servers
makes
sense.
Oh
my
gosh,
this
is
going
to
be
a
win.
We
got
it,
we
did
it.
We've
already
got
all
these
primitives
built
into
the
kubernetes
api
and
let's
use
them.
The
problem
is
that
there's
a
bunch
of
problems
with
Federation?
A
First
of
all,
for
security
as
we're
researching
this,
you
know
our
customers
are
running
these
enterprise-grade
workloads
and
you
you
care
about
this,
and
so
the
way
that
the
API
server
works
is.
It
has
root
on
a
cluster,
so
when
you
move
that
up
a
level
on
to
talk
into
multiple
clusters,
now,
if
you
own,
the
Federation
API
you've
got
root
on
all
the
clusters.
This
is
perfect:
you're,
not
just
bit
coining
on
one
cluster.
You
actually
can
do
it
across
all
30
clusters,
for
example.
A
Clearly
this
is
a
problem
and
then
the
way
that
the
Federation
API
server
is
implemented.
It's
just
the
control
plane.
Basically,
and
so
this
creates
a
single
point
of
failure
in
the
most
common
scenarios,
and
things
like
SED
aren't
designed
to
run
across
a
LAN,
for
example,
and
so
now
you're
gonna
basically
pick
this
entire
tech
stack
the
control
plan
that
you're
used
to
and
it's
got
to
run
somewhere
and,
as
you
know,
you're
all
running
kubernetes.
This
is
not
trivial.
You've
got
a
clustered
staple
database.
A
You've
got
your
H
a
API
server
and
your
hie
controller
managers
and
then
you've
got
all
the
glue,
the
TLS
and
the
secrets
that
join
it
all
together,
and
so
this
is
a
lot
to
manage.
Hence
why
people
want
a
solution
to
this
problem,
and
this
is
what
it
basically
looks
like
you,
have
your
two
clusters
and
pick
six
of
them
doesn't
matter.
It
turns
out
that
you
know,
because
you
can't
run
a
TD
across
the
land
very
easily.
You
end
up
running
that
entire
control
plane
on
one
of
your
clusters.
A
You
just
pick
one
somewhere,
so
you
now
you've
got
a
single
point
of
failure.
Now,
that's
not
like
extremely
critical
unless
all
of
your
components
when
you're
doing
your
software
delivery
are
now
running
through
this
one
cluster.
Clearly,
that's
not
great,
and
you
know
this
could
be
okay
if
you
had
a
better
failover
story,
but
now
you've
got
to
instantiate
a
new
control
plane
on
one
of
these
other
clusters.
Reconnect
all
your
clients
to
it
hook
it
up
to
your
Jenkins
pipelines.
A
All
this
stuff
starts
become
very,
very
complicated,
and
so
we
didn't
really
think
that
Federation,
in
its
current
form,
was
the
way
to
go,
and
what
we
decided
to
do
was
really
take
a
step
back
and
break
up
the
problem
into
its
constituent
pieces
and
see
if
we
can
tackle
some
of
those
or
maybe
even
reduce
some
of
the
scope
from
just
doing
magical
Federation.
You
just
shoot
the
same
deployment
at
a
do
new
API,
and
it's
just
going
to
magically
feder
8
all
the
clusters.
A
I,
don't
think
that
actually
works
in
practice,
so
we've
in
tecktonik
focused
primarily
on
these
two
use
cases.
You've
got
an
app
owner.
This
is
somebody
that
you
know
runs
software
in
their
namespace
and
they've
got.
You
know
a
team
of
people
that
also
work
with
them
and
then
you've
got
your
infrastructure
admins.
These
are
the
folks
that
have
to
run
all
these
clusters,
these
10
to
15
clusters.
We
were
talking
about
so
let's
dig
into
what
an
app
owner
cares
about,
so
you've
got
CI
CD.
A
You
want
to
build
containers
and
then
ship
them
out,
because
your
business
makes
money
when
these
containers
are
running,
and
so
you've
got
to
do
things
like
hook
it
up
and
get
the
secrets
all
configured.
Then
you've
got
to
do
some
cluster
discovery.
If
you've
got
five
clusters,
where
are
they
running?
You
know,
because
the
an
infrastructure
admin
might
be
changing
those
locations
on
you
and
then
should
something
happen.
A
You've
got
a
failover
between
those
clusters,
and
you
know
this
has
a
bunch
of
application,
specific
problems
and
kind
of
sequences
that
you
need
to
do
so.
That's
a
problem
that
you
have
to
own
and
then
there's
a
credential
management
in
general
of
just
rotating
secret's,
regenerating
TLS,
all
that
good
stuff
making
sure
all
this
stuff
does
expire,
but
you've
got
to
be
on
top
of
it
before
it
brings
everything
down
on
the
infrastructure
admin
side.
You've
got
to
track
all
these
clusters.
A
Now
you've
got
10
to
15
clusters,
maybe
you're
running
hundreds
of
clusters,
so
you
care
about
the
security
of
those
you
care
about.
What's
going
on
and
so
you're
responsible
for
that,
because
the
application
owners
are
depending
on
you
to
provide
that
base
functionality
for
them
to
deploy
all
their
applications
and
security
kind
of
goes
from
the
bottom
up.
So
of
course,
and
you're
an
emperor
I
mean
you,
don't
trust
any
of
those
application
teams.
A
So
you
want
to
work
within
the
confines
of
what
kubernetes
gives
you
for
our
back
and
all
the
primitives
that
help
you
lock
down
this
stuff
and
then
resource
limits.
Is
your
last
thing.
As
you
know
we
found
with
our
customers
they
would
deploy
their
first
or
second
kubernetes
cluster
in
their
environment
and
it's
really
great
and
becomes
really
popular
because
from
team
to
team
it
kind
of
spreads.
Oh,
my
gosh
we've
got
this
new
infrastructure
environment.
It's
great
I!
Don't
have
to
worry
about
this
I,
don't
worried
about
that.
A
I
can
scale
easily,
and
so
you
get
a
request
for
another
name
space
for
another
team.
This
team
needs
more
RAM,
and
so
you
have
to
care
about
that
stuff
because
you
can
bring
down
everybody.
If
you
know
you
have
resource
exhaustion
and
kubernetes
has
great
primitives
built
in
for
this,
which
is
the
great
news,
but
you've
got
to
keep
those
synced
up
across
all
of
your
amazon
regions,
for
example.
A
A
Credential
management
and
security
is,
you
know,
should
be
shared
by
everybody,
and
so
how
can
we
tackle
this
problem
so
that
it
meets
the
needs
of
both
of
these
groups?
And
the
answer
is:
we've
got
a
lot.
We
can
lean
on
in
kubernetes
to
do
this.
You
know
you're
all
familiar
with
all
the
the
things
Network
policy
are
back.
All
that
stuff
is
built
in,
but
can
we
just
make
it
a
little
bit
smarter?
A
So
it's
easier
to
manage
and
I
want
to
talk
through
some
of
the
pieces
that
we
have
to
build
with
today
and
if
we
were
going
to,
you
know
not
just
slap
an
API
server
and
call
it
Federation,
but
if
we're
actually
gonna
build
a
piece
of
software,
what
are
the
components
that
we
have
to
do
that?
The
first
one
is
the
cluster
of
registry.
Now
this
isn't
that
crazy?
It's
it's
a
spec
for
an
object.
So
if
you
see
coop
cuddle
get
clusters,
you
can
list
clusters
and
those
have
attributes.
A
This
is
modeled
as
a
CR
D
in
our
system
and
then
there's
a
cluster
registry,
a
piece
of
software
that
we
actually
didn't
consume.
We
were
just
carrying
about
the
modeling
of
this
object.
How
do
I
say
this
is
a
cluster
it
has
these
names.
Has
these
attributes
has
these
labels?
Then
it's
located
at
this
address,
so
you
can
do
this
today
in
kubernetes.
The
problem
is
that
it
runs
as
a
single
cluster.
You
have
to
submit
all
of
your
cluster
definitions
to
every
one
of
your
clusters
and
remember
our
number
of
clusters
is
dynamic.
A
It's
constantly
changing.
So
what
do
we
do?
There?
We've
got
to
get
this
probably
on
all
the
clusters.
If
you
want
software
to
be
able
to
read
and
know
things
about
other
clusters
in
the
environment,
we
talked
about
this
before
access
control.
We've
got
all
these
folks
and
they
need
different
levels
of
access.
You've
got
your
app
owner
who
should
be
able
to
administrate
it
a
namespace
and
be
able
to.
You
know
how
to
remove
people
and
role
bindings
and
that
type
of
stuff
you've
got
an
engineer.
A
Who's
you
know,
probably
has
lower
permissions
they're,
just
shipping
software,
that's
what
they
care
about
on
the
infrastructure
owner
side,
you've
got
your
sres
that
need
to
have
pretty
wide
access
to
stuff
to
be
able
to
debug
issues,
and
then
you've
got
in
at
least
a
proper
setup.
You've
got
all
these
robots
that
are
doing
stuff
on
your
behalf.
These
are
your
Jenkins
workers.
A
You've
got
all
kinds
of
scripting
environments,
doesn't
matter
what
it
is,
and
you
want
them
to
have
as
little
access
as
possible
to
just
deploy
this
single
application
that
they're
working
on
and
nothing
else,
and
then
you
want
to
hook
all
those
folks
up
to
staging
and
production
or
different
environments.
Your
different
clusters,
whatever
it
is,
and
what
does
this
there's?
Is
there
a
magic
piece
of
software
that
does
this
I?
Don't
think
it
exists?
A
Yet
you
know
Federation
can
build
on
some
of
these
things,
but
you
don't
have
the
the
smarts
that
you
need
yet,
and
so
you
need
to
process
this
with
something
and
we
need
to
figure
out
what
that
piece
of
software
is
another
really
nice
thing
about
Federation,
which
I
think
why
it
really
took
off
and
why
you're
all
here
today
is
because
it
allows
you
to
build
your
own
workflows
on
the
api's
that
it
provides.
So
you
know
you,
you
don't
want
to
just
run
off
one
cluster.
A
You
want
to
run
off
several
and
you
want
to
have
all
these
folks
access
to
all
those
clusters
so
having
that
customization
is
key,
but
the
problem
with
Federation
today
is
it's
kind
of
slapped
together
to
be
honest
with
some
annotations.
So
these
are
way
that
you
can
extend
kubernetes
and
that's
all
good
and
great.
But
here
you
can
see
that
you
can
set
some
stuff
to
rebalance
either
true
or
false
awesome.
A
And
then
you
can
say
min/max
on
this
cluster
in
that
cluster
and
so
you've
got
dynamic
resource
scaling,
type
stuff
going
on
there
and
you're
controlling
your
cost.
And
that's
really
it.
You
know
you
don't
have
a
lot
of
really
rich
primitives
for
what
to
do.
If
you
want
to
have
business
logic
in
here
about
how
you
roll
out
to
different
environments,
that
doesn't
exist,
so
you
you
need
to
build
that
workflow
on
top
anyways.
A
So
let's
take
a
few
examples
and
talk
through
how
you
would
do
this,
so
you've
got
your
robots
that
are
building
your
containers
and
shipping
them
off.
But
maybe
you
want
to
have
a
human
in
the
loop.
This
is
like
kind
of
a
chat,
ops
approach
or
whatever
you
want
to
take
that
says.
Yes,
this
package
of
features
is
ready
for
staging
Rev
that
a
few
times
and
say
all
right.
All
nine
of
those
things
now
can
make
it
in.
Let's
build
that.
A
Where
do
you
do
that
to
say,
roll
out
to
an
off
peak
environment
first
and
then
go
to
a
non
peak
environment
if
it's
successful
and
you
can
encode
all
those
rules
in
a
software,
but
that
software
needs
to
exist,
and
this
is
how
we
started
thinking
through
the
problem?
Is
you've
actually
got
two
different
sets
of
users
here?
That
really
need
to
do
two
different
sets
of
things,
and
you
might
need
two
different
sets
of
software
to
do
that.
A
You've
got
your
application
owners
who
are
shipping
out
their
new
versions
and
changing
secrets
and
changing
their
configuration
values
specific
to
their
namespace,
and
then
you've
got
your
operations.
Folks
that
have
this
baseline
set
of
needs
where
they're
managing
our
back
across
all
of
your
clusters.
A
That's
the
object
that
says
this
is
an
IP
I
server
and
this
is
its
location.
This
is
its
name.
It
has
these
attributes
and
we
were
really
using
that
as
a
doing
label.
Queries
across
those
four
policies
and
the
policy
is
really
the
the
bulk
of
this
feature.
And
what
that
looks
like
is
there's
an
example
up
here
on
the
screen
and
it's
to
do
our
back
really
an
access
control
only
across
a
number
of
clusters
and
to
implement
that
there's,
a
small
agent
running
on
every
cluster.
A
That's
talking
back
to
the
central
source
that
holds
these
policies
and
you
map
a
policy
and
a
set
of
clusters
together
and
say
over
here.
We're
saying
on
every
Amazon
environment
I
want
to
have
these
cluster
roles
and
these
bindings
on
these
namespaces
for
these
groups.
So
if
you've
got
LDAP
running
between
all
these
clusters,
for
example,
you
add
somebody
to
a
group
there.
They
have
access
to
all
these
things.
Use
an
admin,
have
one
place
where
you
can
edit
this.
A
You
can
audit
it
security
folks,
love
this,
because
you
know
they're
all
in
one
place,
then
you
can
put
our
back
on
these
policies
once
again
already
built
into
kubernetes.
So
you
get
these
really
rich
workflows
that
you
can
build
around
access
control
and
the
the
part
of
this
that
was
really
specific.
Is
we
didn't
want
to
touch
workload
api's
at
all
deployments?
Replicas
sets
all
that
stuff.
A
That's
on
that
top
side
of
the
cluster,
that's
on
the
the
app
owners
and
their
engineers
to
figure
out
what
that
workflow
looks
like,
and
this
is
just
really
to
be
very
realistic
about
how
we
could
ship
this
in
the
product
and
the
sync
policy
is
great
because
all
we're
doing
is
taking
these
cluster
objects
and
syncing
them
across
all
the
clusters
and
then
we're
taking
the
policies
and
syncing
those
across
the
clusters
too,
and
so
you
phone
in
with
a
cluster
and
say
hey.
This
is
my
name.
A
Send
me
all
the
things
I
need
to
do
it's
a
running
on
a
desired
State
Loup,
just
like
the
rest
of
kubernetes,
so
it
really
makes
sense
to
admins,
and
we
think
that
is
a
really
sane
way
to
operate.
So,
let's
walk
through
some
scenarios
of
what
this
looks
like
in
day-to-day
practice.
We've
got
our
three
environments
here,
so
we've
got
our
cluster
registry,
that's
tracking
all
three
of
the
clusters
and
then
we've
got
our
common
set
of
our
back
on
the
bottom
and
we're
in
Europe.
A
A
If
you
were
deploying
to
this
environment
or
in
this
set
of
clusters-
and
you
want
to
say
okay,
this
thing
is
ready
to
go
to
production.
This
is
what
that
would
look
like.
You
would
have
this
piece
of
software
that
you
are
submitting
manifest
to
and
it's
gonna
then
talk
to
the
cluster
register.
You
can
imagine
doing
this
in
like
a
Jenkins
pipeline
and
say
deploy
this
to
every
production
environment,
whether
that's
three
or
five
or
ten
right
now,
and
it
goes
and
does
that
and
then
on
the
ops
side.
A
A
So,
as
an
admin,
you
go
boot
up,
your
cloud
formation
run
your
installer,
whatever
it
is
and
boom
you
get
your
new
namespace,
because
your
policy
is
now
on
a
label
selector
not
on
a
specific
set
of
clusters
that
production
namespace
can
be
instantiated
immediately
once
that
cluster
is
up
next
time.
Your
Jenkins
job
runs
because
it's
using
this
registry
and
not
a
static
set
of
locations,
you
can
deploy
to
it,
and
you
know
imagine
you
have
some
software
to
wire
up
DNS
and
all
that
type
stuff.
A
A
We
need
to
really
solve
that
security
problem
of
Federation,
where,
if
you
own
it,
then
you
have
root
on
all
the
clusters,
and
so
what
we
decided
on
doing
is
having
a
read-only
poll
model
we're
one
of
these
clusters
as
deemed
the
source
for
the
policies
at
any
given
moment.
This
is,
you
know,
typically
stays
the
same
cluster,
but
all
these
clusters
look
the
exact
same
they're
all
running
this
agent,
and
it
reads
from
the
registry
and
says
you
know
like
this:
cluster
will
check
in
with
its
service
account
and
say:
I.
A
They
can
only
read
clusters
and
cluster
policies
so
already
right.
There
you've
reduced
your
blast
radius.
So
a
these
credentials
can
be
revoked
on
a
cluster
by
cluster
basis,
and
then
they
can't
even
read
from
any
of
the
good
stuff,
which
is
all
the
applications
you're
actually
running.
They
can't
you
know,
modify
any
of
your
auerbach
stuff.
It's
all.
You
know
they
can't
create
new
service
accounts
and
that
type
of
thing-
and
so
the
really
nice
thing
about
this
is
in
a
failover
scenario.
A
All
these
clusters
look
the
exact
same,
and
so
you
have
this
agent,
you
just
flip
it
config
that
says:
I
am
now
the
source
of
this,
because
we
were
already
doing
all
this
syncing.
It's
already
got
the
current
state
of
the
world
and
then
that
european
cluster
could
go
down.
You
SE
starts
reading
from
the
registry,
so
that
your
failover
story
is
not
boot
up
a
new
three
node
at
CD
cluster
and
configure
all
these
things,
they're
already
kind
of
running
a
sort
of
an
active,
active
type
model.
A
So
the
question
is:
where
do
we
go
from
here?
These
components?
You
know
some
of
this
software
needs
to
be
written.
We
started
exploring
some
of
this
in
tectonic,
and
so,
if
you're
going
to
do
this
today,
what
do
you
need
to
do?
And
how
do
you
start
taking
advantage
of
the
cluster
of
registry
and
all
these
other
primitives
that
we
have
to
build
on?
A
And
so
let's
walk
through
them
from
least
complex
to
most
complex,
but
with
the
understanding
that
you
were
going
to
need
some
specialized
software
to
do
this
to
encode
that
business
logic,
it's
not
just
as
simple
as
putting
the
Federation
API
on
top
of
our
clusters
and
calling
it
done.
I
think
we've
kind
of
all
explored
this
and
it
just
doesn't
really
work.
A
So,
let's
start
with
some
very,
not
complex
examples.
The
simple
bash
script!
Everybody
loves
this.
This,
you
know
is
a
simple
loop
through
your
cluster
say
you're,
you
know
you
can
read,
you
can
say
coop
cuddle,
get
clusters
and
then
for
each
of
those
things
throw
it
in
a
loop.
But
you
know
this
isn't
sanchita
by
human,
and
you
know,
maybe
that's,
okay,
for
your
organization,
if
you
don't
deploy
very
often
or
just
prototyping,
something
you
know,
run
the
bash
script
on
your
local
machine
if
you've
got
credentials
to
all
these
clusters,
it's
great.
A
But
what
happens
when
something
goes
wrong.
What
happens
if
you
run
out
of
quota,
for
example-
and
you
can't
make
all
your
pods
well,
there's
no
error
handling
here
and
yeah.
You
can
make
your
your
script
more
complex
and
add
all
these.
If
statements-
and
you
know
really
get
crazy
with
it-
and
you
know-
maybe
that's
fine
too,
but
it's
not
like
that
declarative
state
that
you're
used
to
in
kubernetes
so
now,
you're
kind
of
tweaking
the
way
that
you
think
about
this.
A
When
you
know
why
can't
we
think
of
all
of
these
things
the
same
and
then
maybe
you
want
to
get
a
little
bit
more
automated
with
it.
So
you've
got
your
Jenkins
job
and
so
for
every
PR
to
some
repo
that
holds
all
of
your
manifests.
For
example,
you
do
the
same
thing:
you
loop
through
all
the
clusters,
but
now
the
humans
out
of
the
loop
you're
you're
automating,
some
of
this
and
that's
great,
but
you
have
some
of
the
same
problems.
A
If
something
goes
wrong,
Jenkins
can't
magically
fix
that
it
just
bails
out,
and
so
your
software
doesn't
get
deployed
and
a
human
has
to
go
back
in
and
see
what
happened.
And
then,
if
you
know
you
might
not
have
access
to
look
at
the
resource,
quotas
or
change
the
resource
quotas,
and
so
you've
got
to
go
back
and
forth
with
your
teams.
And
meanwhile
your
pipeline
is
stuck
here.
So
you
know,
maybe
that's
not
great.
Another
thing
is,
if
you
have
to
model
some
business
logic
inside
that
says,
you
know
talking
about
our
off-peak
hours.
A
Now
your
job
is
running
for
a
really
long
time.
Even
if
your
deployment
takes
really
long
time,
then
your
Jenkins
worker
dies
and
your
mid
deployment.
You
don't
know
what
to
do.
You
don't
have
that
declarative
state.
So
nothing
knows
what
to
do
it's
just
in
the
state
that
it's
in.
So
that's
just
really
not
that
great
and
we
know
kubernetes
is
smart,
but
these
solutions
aren't
you're
dumbing
down
your
deployment
pipeline,
but
the
cluster
is
really
smart.
A
So,
let's
get
to
a
more
complex
example.
This
is
not
necessarily
to
do
with
like
deploying
workloads.
This
is
a
very
specific
example
for
ingress.
So
Google
has
this
really
great
demo
of
multi
cluster
ingress
I.
Believe
there's
a
talk
actually
this
afternoon
on
it
if
you're
interested
and
the
point
that
I
want
to
make
with
this-
is
that
it's
using
some
specialized
software
on
the
right
here.
You
can
see
that
oh
we've
got
a
new
CLI
tool.
A
This
isn't
coop
coddle
anymore,
but
it
looks
very
much
like
it
and
it's
actually
using
kubernetes
tooling,
so
you
can
see
there's
a
reference
to
a
coop
config
there.
So
it's
starting
to
get
into
the
kubernetes
ecosystem
feel
a
little
bit
more
native,
but
then
it's
also
using
some
GCP
features.
You
know
it's
really.
It's
a
tech
demo,
I,
don't
know
what
their
plans
for
this
are,
and
the
other
thing
is
that
now
you're
connecting
this
coop
native
tool
to
some
smarts
that
are
outside
of
the
cluster,
which
is
great
if
you're
a
GCP
user.
A
But
if
you're
not
you
know,
this
is
not
portable
across
environments
and
that's
the
whole
point
of
this.
You
know
we
want
that
hybrid
story
and
just
for
fun.
I
actually
was
looking
through
the
documentation
for
this
and
look,
we
see
one
of
these
bash
scripts.
This
is
the
most
simple
version
of
that
is
for
every
context
in
here
go
do
something.
A
Now.
This
next
topic
is
really
near
and
dear
to
my
heart.
This
is
a
this
operator
pattern,
something
you
heard.
Brandon
talk
about
at
the
keynote
we've
been
exploring
this
through
core
OS
since
2016,
and
what
this
is
is
bringing
all
those
smarts
that
we're
just
talking
about
where
it's
outside
of
the
cluster
and
the
multi
cluster
ingress
and
bringing
it
into
the
cluster.
So
an
operator
is
really
a
specialized
control.
A
Loop,
that's
running
just
like
the
controller
manager
in
kubernetes
running
all
of
the
default
primitives
and
it's
listening
on
some
si
RDS
to
do
some
sort
of
logic,
and
this
can
be
anything
that
you
want.
That's
the
beauty
of
it
as
you
can
bring
all
that
business
logic
and
give
it
a
place
to
live
and
because
it's
using
CR
DS,
it's
a
and
kubernetes
extension
point,
so
you're
using
cube
native
tooling
and
then
you've
got
that
desired
state.
This
starts
to
feel
very
nice.
A
It's
very
similar
to
how
you
would
write
your
deployments
and
here's
an
example.
On
the
right
hand,
side
I've,
just
something
I
made
up,
but
this
is
a
compliance
database.
This
is
a
database,
but
it
knows
about
some
legal
issues
that
we
have
with
data
these
days.
So
over
here
I've
got
my
spec
I.
Can
scale
it
up
and
down?
We
want
to
make
sure
that
some
of
this
stuff
is
replicated
twice.
A
Let's
auto
scale
it
based
on
how
much
traffic
is
going
in
and
out
of
the
cluster
back,
it
up
sure
sounds
great
and
then
at
the
bottom
here
we've
got
this
spec
for
geography
and,
let's
just
suppose
in
this
hypothetical
situation
that
you're
storing
some
data
that
needs
to
stay
in
the
EU,
but
maybe
you
would
prefer
to
be
in
Germany
just
because
your
other
services
are
mostly
in
Germany.
But
if
there's
not
a
German
cluster
using
the
cluster
registry,
then
we'll
deploy
it
somewhere
else.
A
So
you
can
see
here
that
the
power
of
this
operator
is
that
this
spec
field
is
not
very
long,
but
you
can
imagine
all
of
the
smarts
that
are
powered
by
this
back
up
hourly
equals
true.
That
sounds
magical,
but
you
know,
there's
like
this
operator
could
have
logic
for
how
to
find
where
the
object
store
is,
for
example,
that
wants
to
talk
to
the
secrets
you
could,
even
in
here
point
at
a
specific
secret.
It
could
have
one
hard-coded,
your
location,
hard-coded.
A
That
type
of
thing,
so
you
really
get
to
see
this
business
logic
that
is
embedded
in
the
operator
and
that's
really
really
powerful.
Now
the
nice
thing
about
this,
too,
is
because
it's
cube
native
you're
using
your
are
back
primitives.
So
imagine
the
you
know:
common
are
back
syncing
that
I
just
talked
about
in
tecktonik.
Now
you're
doing
this
for
all
of
these
customized
operators.
A
The
caveat
this
requires
a
kubernetes
native
application,
I
think
you're
all
here,
because
you
have
those
so
that
shouldn't
be
much
of
an
issue,
but
you've
got
to
have
stuff
that
can
be
containerized,
and
you
know
the
operator,
you
can
imagine
it
wiring
up
all
kinds
of
things
start.
These
deployments
create
these
secrets.
Read
off
this
config
map
set
up
these
services.
Wait
for
this
health
check
to
pass
before
this.
A
Does
this
other
thing
all
that
complex
stuff,
that
you
have
that
usually
embedded
in
either
like
a
run
book,
or
maybe
it's
a
bash
script
that
you
manually
run
or
it's
just
in
somebody's
head?
You
can
model
all
that
in
software
and
the
cool
thing
about
this.
Is
it's
not
just
a
single
cluster
you
can
now,
because
you
have
access
to
this
cluster
registry.
You
could
have
an
operator,
that's
doing
all
kinds
of
stuff
I
mentioned
earlier.
Wiring
up,
DNS
I
was
just
talking
to
somebody
about
an
example
of
you
know.
A
This
could
easily
be
talking
to
any
sort
of
DNS
provider
to
start
wiring
up
things
once
the
application
is
done.
If
you
need
to
failover
and
rebalance
data
before
you
remove
it
or
something
like
that,
you
could
do
any
anything
like
that
and
you
have
it
running
in
the
cluster.
So
it's
got
all
of
the
cluster
state,
which
is
really
powerful
I've
going
on
for
that,
we've
got
a
little
bit,
there's
a
bunch
more
examples.
A
A
The
community
has
obviously
identified
these
and
is
working
to
fix
them,
and
so
I
just
want
to
talk
to
that
for
a
second,
and
the
most
important
thing
is
that
the
special
interest
group
has
kind
of
stated
that
one
of
their
goals
is
that
we
don't
want
to
tackle
this
entire
problem
space,
because
it's
really
really
wide.
As
I
talked
about
with
you
know,
you've
got
all
these
different
needs
for
these
different
users.
You've
got
different
environments,
different
clusters.
You've
got
to
track
all
this
stuff,
and
maybe
somebody
wants
to
model.
A
You
know
something
that's
operated
and
then
you've
got
a
placement
modifier
which
says
when
you
instantiate
this
template.
Where
does
it
go?
And
this
is
a
label
selectors
and
all
that
kind
of
cube
native
stuff,
then
you've
got
an
override
object
that
you
can
submit
into
very
specific
clusters
if
you
want
to,
for
some
reason,
only
scale
up
one
cluster
outside
of
what
the
scheduler
is
doing
say
bump
this
limit,
where
I
want
I
need
five
of
these
running.
A
Just
right
now
just
make
it
happen,
but
you
can
actually
override
any
value
in
the
template
which
is
really
powerful.
So
if
you
need
to
change
a
the
labels
that
one
of
the
objects
has
in
just
a
specific
cluster,
this
is
what
you
use
those
overrides
to
do,
but
it
could
be
anything
so
this
is
where
you
could
start
doing
the
locality
based
stuff,
where
I
need
to
override
Germany
and
put
a
different
country,
for
example.
A
This
is
the
model
that
Federation
v2
is
going
to
work
on
nice
thing
again,
it's
kubernetes
native,
so
you
can
use
all
the
are
back
primitives
that
you
already
have
today,
and
then
you
have
the
nice
primitives
to
push
it
into
like
a
policy
engine.
If
you
won't
need
to
do
auditing,
if
you've
got
a
security
group
that
really
wants
to
keep
a
close
eye
on
things,
there's
a
number
of
projects
out
there
that
are
starting
to
bootstrap
into
this
space
too.
So
because
we
have
these
really
nice
objects
to
work
with.
A
The
UX
of
this
matters
and
I
think
I've
mentioned
the
word
kubernetes
native
or
kubernetes
tooling
a
lot.
This
is
really
important,
because
this
is
the
way
that
folks
expect
to
interact
with
clusters,
because
it's
what
they
do
in
their
day
to
day
job
you're,
you're,
changing
around
deployment,
values
and
config,
Maps
and
generating
secrets
and
you've
got
that
whole
workflow.
It's
very
understood,
and
that
means
that
you're
then
portable.
You
do
have
that
hybrid
story,
if
you're
only
working
against
kubernetes
api
x'
under
the
hood.
This
is
implemented
as
an
aggregated
api
server.
A
Add
your
own
code
would
definitely
love
to
see
that
and
then
some
things
that
are
also
possible
as
the
next
steps
for
this
you
could
start
implementing
a
custom
scheduler
for
this,
just
because
it's
plugging
into
kubernetes
and
it's
also
secured
with
service
accounts
much
the
same
way
that
our
tectonic
multi
cluster
is
so
once
again
Kubb
native
using
that
are
back
we're
building
on
these
pieces.
So
we
don't
have
to
you
know,
have
this
huge
problem
space
we're
condensing
it
down
into
something:
that's
a
little
bit
more
manageable.
A
So
the
takeaway
that
I
want
you
to
have
today
is
that
this
problem
space
is
going
to
require
custom
software
and
we
just
walk
through
a
number
of
ways
that
you
know
a
bash
script
is
custom
software.
It's
pretty
attainable,
you
can
go
home
and
do
this
today
and
so
I
want
you
to
start
thinking
about
that
as
you're
planning,
for
what
you're
gonna
do
post
Federation,
because
right
now,
Federation
is
not
a
thing.
The
project.
A
You
know
it's
going
to
get
retooled,
but
that's
going
to
be
a
little
bit
and
if
you've
got
five
or
ten
clusters
that
you
want
to.
You
know
work
with
today
and
you
start
writing
some
specialized
software
or
at
least
start
thinking
about
how
your
business
rules
get
modeled
and
some
sort
of
logic
and
choose
things
that
are
Kubb
native.
A
The
best
user
experience
is
when
you're
using
coop,
tooling
everybody's
already
used
to
all
those
primitives,
and
then
you
get
the
power
of
kubernetes
is
when
you're
building
off
of
all
these
primitives
that
desired
state
loop
is
just
killer.
This
is
why
kubernetes
is
so
popular.
It
just
makes
sense.
It
works.
You
can
write
software
to
do
things
that
software
is
good
at
and
let
humans
do
things
that
humans
are
good
at
so
take
use
the
power
of
those
explore
the
registry
and
all
that
and
then
go
home
and
make
some
software.
A
You
can
do
all
this
today
start
it's
not
bad
to
have
a
bash
script
first
and
then
you
know
move
to
a
Jenkins
job.
Maybe
you
want
to
explore
using
an
operator.
Maybe
you
want
to
explore
some
of
the
other
tooling
out
there
that's
more
specialized,
but
you
can
do
this
today
and
it's
definitely
attainable
and
then
lastly,
get
involved
with
Federation
v2
you're
all
here,
because
you're
interested
in
this
problem
space
and
like
the
kind
of
charter
for
this
group
says,
is
we're
not
going
to
solve
all
these
problems.
A
So
y'all
can
invent
different
pieces
that
can
solve
this,
and
you
know
every
Morgan
ization
is
different.
Everybody
models
things
differently,
but
we
need
those
view
points
to
make
this
the
best
that
it
can
be
to
add
the
right
extension
points
so
that
you
can
use
the
primitives
but
then
build
on
top
of
it
and
that's
all
I
have
thank
you.