►
Description
Multi-Cluster Ops in a Hybrid World [A] - Vitaliy Zinchenko & Kire Filipovski, Oracle
The reality of multiple Kubernetes deployments typically leaves you with varied cluster profiles, deployed on a mix of on-prem and public cloud environments. Production ops for large distributed systems is hard enough in a single environment, but becomes even more complex with hybrid conditions.
In this talk, we’ll dissect how to leverage federation for Kubernetes governance across capacity management, micro service dependencies, infrastructure upgrades, versioning, and security, as well as, global high availability, continuity, and resiliency, in a hybrid environment.
About Kire Filipovski
Kire Filipovski works as a Cloud Architect at Oracle leading design and implementation of a distributed containerized application management system. Previously Kire worked as a Distinguished Cloud Architect at Walmart where he designed computing platforms that transformed the world's largest company into a digital business. He joined Walmart in 2013 as part of the OneOps acquisition, a company which he co-founded in 2011.
About Vitaliy Zinchenko
Vitaliy Zinchenko is Oracle’s Cloud Architect working on the design and implementation of a Global Application System for Oracle Cloud customers. Prior to joining Oracle, Vitaliy was with Walmart Labs as a Principal System Engineer, where he implemented a cloud based application lifecycle management system that brought a developers and operations experience to the next level. He joined Walmart in 2013 as part of the OneOps acquisition, a company which he co-founded in 2011.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Another
obligatory
slide
about
you
know
kubernetes
preaching
to
the
choir
here,
kubernetes
growing.
This
is
from
the
survey
from
the
spring
production,
picking
up
really
bringing
up
the
topic
of
multi
cluster
more
and
more,
especially
as
production
workloads
going
there
take
your
pick.
Probably
one
of
these
is
one
of
the
reasons
why
you're
looking
at
multi
cluster
as
well,
but
generally
the
train
is
going
there.
A
What
happens
when
you
do
multi
cluster
lots
of
challenges
to
look
at?
We
try
to
look
at
a
little
bit
from
a
different
angle,
a
little
bit
earlier
up
the
decision
stream.
If
you
will,
in
terms
of
how
you
arrive
to
proper
multi
cluster
strategy,
so
literally
a
stream
of
decisions
that
lead
to
certain.
A
You
know,
environments
where
you
have
to
fulfill,
but
basically
the
type
of
questions
that
come
in.
Usually
it's
well
how
many
clusters
do
I
run?
What's
the
best
practice,
there
do
I
run
cluster
for
every
business
unit,
for
every
app
for
every
service
for
environments,
for
all
these
questions
usually
come
up.
When
you
start
looking
at
a
real
live
deployment
of
kubernetes
it
with
multi
cluster
right,
then
you
get
into
you
know
how
big
the
cluster
should
be
when
do
I
resize
them.
How
fast
do
I
make
those
decisions?
A
What
type
of
tech
do
I
use
to
manage
the
lifecycle
of
those
clusters?
When
is
that
decision
made?
What
providers
do
I
use
if
you're,
if
you're
running
on
public
or
even
in
private,
in
terms
of
what
type
of
tech
stack
you
use?
You
know
how
to
guard
myself
against
some
of
these
decisions
in
the
future
to
avoid
potential.
You
know
day
2
problems
or
whatever
they.
You
know
later
down
the.
A
A
Yeah
sure
this
it's
a
very
generic
way
of
representing
but
normally
would
be
some
kind
of
organizational
solution
to
those
questions.
With
a
lot
of
processes
and
tools
out
there
you'd
usually
start
with
some
kind
of
decision
and
design
to
containerize
an
application
to
use
govern
it
as
from
there.
You
would
meet
and
you
do
some
kind
of
reviews
about
what's
current.
A
What's
the
latest-
and
you
know
those
decisions-
and
you
know
they're
delegated
to
other
teams
like
development
and
ops,
to
be
able
to
kind
of
execute
upon
those
decisions,
a
lot
of
communication
going
back
and
forth.
Ultimately,
it
ends
up
with
some
work
being
done
by
the
operations
team,
usually
to
manage
the
clusters
development
to
deploy
those
clusters,
picking
their
own
solutions,
getting
some
reports
out
of
it
closing
the
loop
loop
and
basically,
you
can
end
up
organically
grow
into
this.
A
Usually,
as
you
go,
you
kind
of
end
up
requiring
people
with
certain
skills,
the
better
people.
You
have
the
better
you're
going
to
do
this.
Some
companies
do
better
some,
some
do
it
worse,
but
in
general,
what
we
looked
at
it.
It's
it's
really
a
process
organization,
people
type
of
solution
to
those
questions
that
we
posed
at
the
beginning
right,
we
kind
of
started
looking
at
well,
you
know
the
proverbial.
There
must
be
a
better
way
right,
so
we
kind
of
decided
to
take
a
blank
look,
obviously,
no
better
waiting
indicator.
A
Blank
looked
and
a
blank
slide,
I
guess,
but
literally
we
went
at
some
thing
that
can
use
some
text
some
sulfur
to
be
able
to
tackle
some
of
those
questions
a
little
bit
later.
So,
given
that
we're
committed
to
the
kubernetes
ecosystem,
whatever
the
first
obviously
thing
you
start
looking
at
when
you're
talking
multi
cluster
is
the
kubernetes
Federation.
How
many
of
you
are
familiar
with
the
Federation
all
right,
good?
How
many
of
you
have
tried
a
federation?
A
Alright,
so
a
few
hands?
The
Federation
is
there
to
do.
Multi
cluster,
we
looked
at
it
and
it's
if
you
start
going
through
the
documentation
and
the
set
up
you'll
end
up
doing
something
like
install
a
control
plane.
It
sits
in
its
own
control
plane.
Basically
runs
separate
from
your
clusters:
you
start
up
a
federation,
basically,
normally
what
you
would
do
is
you
would
go
through
and
add
some
clusters.
They
call
it
join
right.
A
A
You
can
use
coop
control,
it's
very
similar
to
deploy
into
a
cluster
but
you're,
deploying
to
this
sort
of
virtual
layer,
which
then
you
know,
delivers
the
necessary
application
to
the
multiple
cluster,
that's
kind
of
how
you
would
end
up
at
this
point,
looking
at
multi
cluster
right.
So
if
you
look
at
this
and
you
compare
it
to
the
type
of
questions
that
we
posted
in
the
beginning-
well,
it's
kind
of
scratching
the
surface
from
some
of
them,
but
a
lot
of
them
are
kind
of
left
on
an
insert.
A
If
you
will
right
so,
it
may
start
giving
you
a
little
bit
of
an
option
of
how
to
deploy
apps
to
clusters,
but
it
does
nowhere
near
in
terms
of
making
decisions
about
how
many
clusters
should
be
out
there,
how
what
the
size
of
the
cluster
should
be.
Who
is
responsible
for
them?
None
of
that
stuff
is
really
covered
with
the
Federation,
but
it's
a
really
good
stepping
stone.
We
looked
at
it.
A
It's
a
separate
control
plane,
it's
a
very
interesting
piece
of
the
puzzle,
so
what
we
started
thinking
about
it
as
well
right
now,
it's
a
little
too
late.
We
already
build
the
clusters
right.
If
you
wanted
to
tackle
those
problems,
what
we
wanted
to
figure
out
well,
what?
If
we
remove
the
clusters?
Has
anybody
tried
this
in
the
Federation
deploying
an
app
without
having
any
clusters
in
it
No?
A
So
the
Federation
has
its
own.
It
city
story,
has
its
own
sort
of
desired
state
and
all
of
that
stuff.
So
what's
gonna
happen
is
the
applications
gonna
go
in
there.
It's
just
not
going
to
be
scheduled
because
there
is
not
it's
akin
to
not
having
nodes
in
a
cluster
right,
so
you're
gonna
say
well,
it's
useless,
but
it's
not
useless.
What
you've
just
did
you
just
captured
the
desired
state
right,
so
we
said
that's
a
great
starting
point
for
us
right.
We
have
a
desired
state.
A
Now
what
we
can
do
is
we
have
an
input
that
we
can
use
programmatically
to
answer
those
questions.
So
if
I
have
an
actual
deployment,
I
basically
have
a
programmatically
way
to
describe
the
demand
from
my
developers
right.
What
I
need
to
do
is
figure
out
how
to
add
something
that
will
derive
the
answers
that
I
wanted.
It's
like
if
I
know
what
they're
trying
to
deploy
I
may
be
able
to
figure
out
what
clusters
doing
the?
How
big
should
they
be?
You
know
it
says.
A
So
what
navarchus
is
is
a
controller
that
sits
within
Federation
control,
plane
that
specifically
listens
to
your
deployments,
and
it
will
start
making
decisions
about
what
clusters
you
need
in
the
whole
world
right.
So
what
does
that
mean?
Obviously,
without
going
into
too
much
details,
we'll
cover
a
little
bit
later
about
the
details,
but
that's
kind
of
the
general
direction
change.
A
If
you
will
right
what
we
did
in
addition
to
that,
we
added
one
extra
component
to
be
able
to
offload
no
markers
for
some
of
the
low
level
sort
of
piping
in
terms
of
managing
the
clusters.
So
we
create
another
component
called
cluster
manager.
So
really
what
happens
is
navarchus
listens
to
the
demand
makes
decisions
of
what
supply
you
need
and
that's
supplied.
Then
it
gets
orchestrated
by
what
we
call
the
cluster
manager.
It's
actually
pretty
simple
wrapper
around
things
like
cops
and
we
implemented
a
wrapper
around
the
Oracle
container
engine.
A
So
what
it's
doing
is
essentially
acting
is
the
human
who
used
to
see
it
doing
the
cops
command
lines
right
now.
The
navarchus
emulates
that
human,
basically,
so
what
you
end
up
with
the
clusters
get
created
and
the
implication
gets
deployed
so,
but
what
we
really
ended
up
with
is
having
this
control
plane,
where
we
can
use
software
to
tackle
a
lot
of
those
questions,
and
that's
where
we
add
right
now,
so
we're
gonna
get
into
a
little
bit
of
the
advantages
of
doing
that.
A
But
what
we
really
call
this
thing
is
an
application
aware
infrastructure.
Ideally
it's
all
about
the
developers
writes
the
app
is
the
king
here
right.
Everything
else
should
really
be
contextually
based
on
the
app.
So
at
this
point,
what
we're
trying
to
do
is
essentially
kind
of
open
up
that
control
plane
to
be
able
to
very
dynamically
procure
adjust
infrastructure
based
on
the
application
needs
we're
gonna,
do
a
really
quick
demo.
Vitaliy
is
going
to
do
a
demo
here.
I'm
gonna
come
back
and
talk
a
little
bit
more
about
the
advantages.
B
With
the
demo,
she
was
what
we're
gonna
go
so,
as
he'd
already
mentioned,
navarchus
runs
alongside
with
kubernetes
Federation,
so
here
I
already
have
my
kubernetes
cluster
up
and
running
with
Federation
parts
provision
you
can
see.
There
is
the
two
parts
for
Federation
API
server
and
Cuba
controller
manager,
and
these
are
a
couple
parts
for
navarchus,
namely
navarchus
and
the
cluster
manager
itself
and
it'll
it
a
little
helper
powder
that
I
will
explain
later
on.
B
B
B
Obviously,
so,
let's
take
a
look
on
these
cluster
configuration
yeah,
most
definition,
so
it
basically
the
generic
Federation
cluster,
although
with
some
additional
configure
tributes
that
are
required
by
the
cloud
provider,
so
standard
stuff,
like
vulnerability
zones,
working
out
shapes
workers
for
abilities
on
and
so
forth.
That's
what
it
looked
like
for
o
for
Oracle
and
these
are
for
AWS,
pretty
much
the
same
thing
just
a
little
bit
more
specific
attributes,
aww
isolated
with
zones,
public
SSH
keys
and
whatnot
that,
and
you
can
see
all
of
those
configurations
are
put
it
here
as
an
annotations.
B
Configuration
objects
into
into
our
Federation
so
again,
I'm
not
joining
any
live
clusters
into
the
Federation
I'm.
Just
configuring,
a
placeholders
that,
if
demand
will
rise,
will
get
provisioned
and
the
cluster
will
become
available
for
the
Federation
to
deploy
to
so
we
created
okay.
So
now
we
got
three
clusters
loaded
into
the
system
and
the
role
in
an
offline
state.
As
going
on
there.
B
So
the
next
step
would
be
to
create
a
demand
that
we're
going
to
have
to
consume
those
those
those
resources.
So
this
is
a
simple
deployment.
I
configure
the
channel
to
request
the
50
replicas,
it's
the
image
on
a
public
dock
repo.
It's
simple,
no
ji
said
that
I
just
want
to
go
ahead
and
deploy
into
my
clusters.
So
take
a
look.
What
was
going
on
with
our
cloud
provider?
So
obviously
this
is
Oracle
kubernetes
engine
nothing's
wrong.
B
B
First
yeah
first
I
need
to
create
a
namespace
and
then
I'm
gonna
create
a
dummy
deployment
was
in
this
namespace
right.
So
we
can
see
the
deployment
is
created
with
desired
number
of
replicas
15,
but
nothing
is
currently
running,
but
the
markers
already
detected
that
I've
got
our
ADA
mint
and
going
through
my
available
supplies.
B
B
Again,
as
you
can
see,
this
is
two
different
zones
in
in
a
public
outcry
at
two
different
providers
and
the
deployment
will
get
deployed
to
all
all
three
of
them.
So
this
is
a
little
a
dashboard
that
we
created
on
top
of
the
Federation.
We
have
a
little
control
around
alongside
that
streams,
the
Federation
events
into
the
elasticsearch-
and
this
gives
us
a
good
possibility
to
visualize.
B
It
will
reduce
tribute,
redistribute
the
paths
across
all
three
of
them
evenly,
so
the
system
can
stop
stream
when
it
turns
the
state
of
the
clusters
and
available
capacity
of
those
clusters
and
based
on
these
data,
it
actually
scheduled
the
application
replicas
accordingly.
So
it's
like
one
level
of
scheduling
the
same
thing.
The
kubernetes
does
across
the
worker
nodes
the
navarchus
does
on
on
the
cluster
level.
So,
as
soon
as
it
will
become
available,
seeing
it's
coming
up
and.
B
Right,
okay,
so
it's
now
we
have
three
clusters
and
Oracle
deployed
this
application
evenly
like
seventeen
parts
in
Yelich.
Seventeen
is
to
seventeen
in
Oracle
and
sanity
and
Westeros.
So
that's
that's
what
we
called
the
just-in-time,
the
kubernetes
cluster
and
again,
if
I,
if
I,
if
I
drop
down
I,
delete
this
this
deployment
and
I
release
resources.
All
those
clusters
will
get
shut
down
and
then
they
will
wait
until
the
next
amount
or
commonplace
all
right.
So
that's
all
kinda
tore
back
to
here.
So.
A
We
do
have
actually
one
more
shorter
demand
later
we're
gonna
try
to
show
a
bursting
use
case
if
we
get
enough
time,
but
I
wanted
to
cover
really
conceptually.
So
what
does
this
mean?
Like
I
mean
that
was
kind
of
explanation
of
how
it
works,
but
I
want
to
go
back
and
sort
of
start
exploring.
Really.
The
benefits
are
the
advantages
of
taking
this
type
of
approach.
A
So
one
of
the
reasons
we
actually
did
it
originally
with
the
Federation
is
to
provide
some
consistency
on
the
clusters.
We
realized
that,
if
you
don't
have
control
over
who's
building
the
clusters,
you
really
don't
have
much
control
over
I
mean.
Does
it
run
ingress
or
not?
Does
it
run
what
type
of
networking
does
it
gets
really
complicated
from
a
Federation
perspective?
A
It's
like
herding
cats
at
that
point
right
so,
for
our
solution
was
well
in
order
for
us
to
run
a
successful,
multi
cluster
deployment,
whatever
we
have
to
get
gain
more
control
over
the
capabilities
of
those
clusters
and
obviously,
since
we're
provisioning
the
clusters,
we
can
ensure
that
they
have
the
right
or
similar
capabilities.
One
way
to
do
it
is
to
kind
of
wait
for
conformance
and
all
that
stuff,
but
since
we
have
control,
we
can
the
consistency
is
key.
There.
A
You
can
run
the
same
port,
you're,
right
kind
of
ingress,
you're,
right
kind
of
add-ons,
the
right
kind
of
C&I
or
whatever
it
is.
You
have
full
control
over
that,
so
that
it's
you
know
it
matches
what
you
really
need
to
do
for
your
applications
right
manageability,
if
you
notice
one
thing
happens,
is
once
you
start
treating
the
clusters
you're,
essentially
going
from
treating
the
clusters
as
pets
to
treating
them
as
cattle,
it
become
very
ephemeral
right
and
that
that
actually
creates
a
lot
more
manageability.
A
In
my
opinion,
because
now
you
can
start
doing
upgrades,
you
can
move
things
much
much
easier
than
before.
Where
you
kind
of
treated
your
clusters.
Is
you
know
your
little?
You
know
static
assets
that
you
wanted
to
use
right.
Obviously,
portability
of
the
application
also
follows
because,
as
you're
moving
clusters,
you
can
move
start
moving
applications
a
little
bit
faster
with
this
stuff,
so
that
it
gives
you
more
portability
of
your
application.
A
Cost
is
a
huge
thing
right
so
because
we're
doing
just-in-time
cluster
we're
essentially
constantly
optimizing
the
available
allocated
infrastructure
that
you
pay
for
to
match
exactly
what
the
current
demand
is.
This
is
important
in
a
public
cloud,
but
you'll
see
that
there's
actually
various
type
of
use,
cases
for
for
cost
and
we'll
cover
in
that
that
actually
do
it
in
on
a
private
cloud
as
well.
A
There
is
for
things
like
governance
and
compliance
I
mean
you
can
do
significant
amount
of
things
in
that
layer
before
your
applications
hit
the
actual
infrastructure
right.
So
if
you
start
thinking
about
security,
compliance
or
or
even
just
budgets-
and
things
like
that
and
I'll
cover
some
of
the
use
cases
later,
but
it
can
improve,
there's
tons
of
compliance
use
cases
that
can
take
advantage
of
that
pattern.
A
If
you
will
right
multi-cloud
managing
you
know
so
one
of
the
things,
for
example,
auto-scaling,
we've
implemented
scale
up
scale
down
in
that
stuff,
but
we're
doing
it
right
into
the
Federation
control
plane
using
an
abstract
object.
What
that
gives
you
is
the
ability
to
scale
up
and
down,
regardless
of
who
your
provider
is,
which
gives
you
a
lot
more
multi
cloud
compatibility,
because
all
the
business
logic
or
operational
logic
that
you're
putting
in
the
system
it's
not
really
specific
to
the
to
any
provider
global-scale.
A
Obviously,
when
you
want
to
light
up
certain
regions,
certain
continents
and
all
of
that
stuff,
it
should
be
fairly
easy
to
just
kind
of
extend,
because
everything
should
really
work
at.
That
point
is
just
matter
of
declaring
the
right
policy
and
the
right
selection
criteria
there
when
you
get
to
that
from
that
central
control,
player
and
obviously
the
ability
to
do
private-public.
A
A
little
bit
easier,
so
a
lot
of
these,
like
you
know
there
is
kind
of
like
it's
almost
like
a
platform
at
that
point,
that
you
can
keep
working
on
and
adding
more
use
cases.
So,
if
you
look
at
some
example
use
cases
that
we
can
solve
obviously
doing
a
continuous
delivery
of
an
application
to
multiple
regions-
and
you
know
we
didn't
talk
too
much
about
how
the
application
is
actually
configured
in
that
multi
region.
A
We've
done
a
bit
of
a
development
on
you
know,
adding
additional
sort
of
federated
ingress
controls
and
stuff
won't
cover
that
a
little
bit
as
well.
But
that's
the
the
sort
of
the
primary
use
case
to
be
able
to
deploy
apps
to
multiple
regions
and
kind
of
be
able
to
kind
of
scale
up
and
down
any
time
you
one
bursting
and
I,
don't
mean
necessarily
bursting
just
in
a
classical
single
app
going
from
private
to
public.
There's
all
sorts
of
interesting
use
cases
that
we
can
do
it
bursting
by
prioritizing
the
cluster.
A
For
example,
as
I
said
in
a
private
cloud,
you
can
have
a
set
of
dedicated
infrastructure
for
certain
business
units
and
a
set
of
shared
infrastructure
for
anybody
to
burst
on.
So
you
should
be
able
to
run
your
base
workload
and
only
when
you
need
it's
going
to
burst
to
a
set
of
clusters
with
a
lower
priority
that
are
shared.
That
way,
you
can
more
effectively
share
private
infrastructure
for
multiple
business
units
or
on
Amazon.
A
If
you
want
to
use
spot
instances
versus
dedicated
instances,
so
you
go
on
the
reserve
they're
there,
but
then
you
can
burst
on
the
spot,
for
example,
so
or
the
demo
we're
going
to
do
is
potentially
if
the
application
allows,
depending
on
latency
and
and
data
affinity,
you
should
be
able
to
burst
from
private
to
public
as
well.
So
all
that
stuff
is
an
option,
because
your
control
plane
does
allow
you
to
do
that
sort
of
like
baked
in.
A
Obviously,
we
talked
about
manageability
the
ability
to
do
you
know
kubernetes
cadence
releases
everybody
stuck
with
doing
upgrades.
Well,
when
the
clusters
are
very
ephemeral,
you
can
inline
upgrades.
You
can
still
do
them,
probably
depending
on
what
your
underlying
take
you
used
to,
to
be
able
to
do,
but
major
upgrades
could
become
a
problem.
This
allows
you
to
simply
junk
a
cluster,
replace
it
with
another
clause,
sir,
without
even
telling
anybody
in
development,
basically
continuous
integration
with
sort
of
temporary
clusters.
You
know
you
deploy
the
app
the
app
deploys,
the
cluster.
A
You
do
an
end-to-end
test
you
on
deploy
the
app
it
shuts
down
the
cluster,
that's
a
use
case
to
and
as
I
said
as
we
realize
that
we
have
this
sort
of
limbo,
control,
plane
that
sits
in
between
we
can.
We
can
start
looking
at,
for
example,
service
meshes.
If
you
want
to
do
inject
things
on
a
global
scale,
you
can
use
that
control
plane,
because
you
have
visibility
for
the
global,
this
sort
of
the
global
desired
state.
A
If
you
will,
we
can
start
using
it
for
for
security
to
insert
network
policies
to
insert
service
mesh
rules,
and
all
of
this
stuff
is
possible.
We've
done
some
pilots.
These
are
kind
of
nodding,
yeah,
something
we
have
implemented
yet,
but
a
possible
use
cases
that
we
can
take
advantage
of
also
managing
budgets.
B
Yeah
sure
yes,
so
for
this
demo
I'm
trying
to
simulate
the
use
case
when
you
have
your
private
cloud
with
software
running
in
it
and
you
want
to
burst
out
into
the
public
cloud,
so
I
had
a
filtration
running
locally
on
my
laptop
on
mini
cube
and
I.
Have
another
cluster
running
again
on
a
laptop
on
mini
cube
and
I
joined
this
cluster
to
my
Federation
to
be
a
managed
cluster
and
I.
Call
this
my
little
private
cloud
and
on
top
of
that
I
registered
my
burst
out.
B
Cluster
object,
I
configured
it
to
be
used
in
Oracle,
cornets
engine
and
usually
whenever
says
cool
around
locally.
My
Oracle
cluster
is
offline.
I
didn't
touch
it,
so
let's
take
a
look
at
how
it
works.
So,
as
I
said,
I
have
two
clusters
here
worked
out:
okie
Oracle,
Cunha
Ascension
is
an
offline
MQ
one
is
in
ready
state
and
let's
take
a
look
a
little
bit
in
more
details.
B
The
the
key
difference
here
is
this:
this
new
cluster
priority
attribute
that
is
for
Oracle,
is
set
to
2
and
4
min
EQ
1.
The
priority
is
1,
and
this
has
a
little
discuss
limitation
number
of
allocatable
for
this
set
to
30
and
the
you
thought
is
already
25,
because
I
already
deployed
my
app
here
and
it's
already
used
up
some
of
the
capacity
so
again
back
to
little
dashboard.
Here
we
see
that
pretty
much
counted.
Percent
of
this
deployment
is
residing
mq
1,
which
is
my
private
cluster,
and
my
burst
out.
B
B
Okay,
yeah
I
just
wanted
to
to
demonstrate
that
this
deployment
have
desired.
Applicants
20
and
what
we're
going
to
do
is
I'm
going
to
scale
it
up
to
14
and
let's
see
what
the
system
gonna
do
in
this
case,
so
undo
the
cube
cradle
scale
and
put
a
replica
sickle
to
14
all
right.
So
it's
scaled
now
and
we
can
see
yeah
that
it's.
B
B
So
as
soon
as
will
become
available,
what
its
gonna
do
is
gonna
fill
up
all
the
available
capacity
on
our
private
cloud
and
whatever
remaining
powers
left
is
gonna
burst
out
into
my
public
lot
faster.
So
we
can
see
the
like
okay,
my
Oracle
clusters
up
and
running,
and
if
you
go
back
here,
this
is
a
distribution,
so
the
blue
one
is
whatever
the
left
number
of
pods
got
burst
out
into
the
public
cloud,
and
it's
used
up
all
the
capacity
on
on
my
mini
cube.
B
B
If
you
go
back
here,
you
see
now
all
the
20
replicas
are
back
on
my
mini
cube
and
on
my
burst
out,
cluster
I
have
zero
versus
zero
Pat's.
On
now
the
system
again
monitors
that
okay,
my
burst
out
cluster
become
a
tidal,
so
pretty
much
no
user
path
around
there
and
within
a
idle
TTL
time.
If
mussels
go
nose
around
it
will
pretty
much
shut
it
down
because
we
don't
need
it
anymore
and
that
basically,
what
we're
going
to
see
here.
B
A
How
come
he
gets?
The
applause
all
the
time,
just
one
more
slide,
we're
done.
We
got
five
minutes
for
question.
We
open
source
this
stuff
yesterday.
Currently
it's
on
the
Oracle
github
account
there's
actually
the
primary
component
there
navarchus
is
there
at
the
cluster
manager
supporting
component.
We
also
made
some
additional
enhancements
and
changes.
We
have
a
federated
in-ears
controller
that
uses
DNS
as
a
back-end.
This
is
has
to
do
with
kind
of
managing
the
application
a
little
better
in
multi
cluster.
A
That
was
not
focused
today,
but
it's
we
got
some
more
code
out
there
that
we
didn't
cover.
Today
we
are
going
to
work
with
the
cig
multi
cluster,
to
see
what
you
know.
If
we
can
kind
of
make
this
an
incubator
project,
we're
just
kind
of
starting
those
discussions
right
now,
but
that's
that's
pretty
much
it
any
questions.
B
A
Yeah
great
question,
so
the
question
was,
like
you
know,
the
underlying
problem
of
all
the
applications
running
into
multiple
providers
with
stateful
sets.
Yes,
so
and
I
kind
of
hinted
a
little
bit
on
that.
Obviously,
the
runtime
needs
to
support
that
now.
What
I
will
tell
you
right
now
is
that
we're
working
on
potentially
doing
a
full
stateful
set
global
application.
Now
the
Federation
has
still
hasn't
released.
A
The
stateful
sets
yet
so
we're
kind
of
trying
to
we've
done
some
pilots,
where
we
can
link
up
the
clusters
using
calico,
where
we
run
like
direct,
so
some
of
the
applications
that
are
stateful,
especially
if
you're
running
your
stateful,
set
as
a
beta
application
within
kubernetes
you'll
be
able
to
use
the
Charlotte
application.
So
essentially
it's
going
to
start
create,
look,
let's
say,
you're
running
Cassandra
or
Kafka,
or
something
like
that
right.
A
It's
going
to
start
sort
of
bringing
its
own
data
as
replicas
into
the
other
providers
to
be
able
to
that's
not
supported
right
now,
but
it's
it's
coming
along
and
obviously,
as
I
pointed
out,
the
bursting
from
private
to
public
in
a
single
application.
It's
a
kind
of
complicated
use
case,
but
that
but
the
burst
thing
as
I
mentioned
it
within
the
same
cluster,
to
be
able
to
share
some
of
the
local
infrastructure,
either
in
a
public
or
private
provider.
A
It's
a
valid
use
case
that
you
can
take
advantage
of
right
now,
while
we're
waiting
for
sort
of
better
support.
You
know
for
stateful
apps
across
our
hybrid,
but
yes,
I
mean
that
that's
an
underlying
problem
that
you
can
solve
with
a
control
point
on
top
of
it
right
you
have
to
be
able
to
we
can.
What
we
want
to
do
is
enable
compatibility
if
the
application
supports
that
that
your
control
plane
will
support
that,
but
it's
a
little
bit
of
a
chicken
in
the
egg
right.
A
A
That's
correct,
the
federated
control
point
is:
is
it's
got
its
own
8
CD?
It
uses
the
same
API
machinery
state
reconciliation
as
the
primary
cluster
right
now.
The
way
they
distributed
it
actually
kind
of
needs
its
own
host
cluster.
But
it
should
sound
completely
up
to
you
on
how
you
run
the
Federation
control
plane
itself
with
navarchus.
Now
we've
been
doing
a
lot
of
work
on
trying
to
create
highly
available
federated
control
plane
because
it's
kind
of
becomes
the
critical
point.
A
So
we're
probably
going
to
come
up
with
some
solutions
soon
that
you
look
into
it.
But
but
yes,
it's
it's
its
own
control
plane
at
this
point-
and
you
know
obviously
I-
would
expect
that
at
some
point
you
know
either
you
can
run
that
control
plane
on
your
own,
with
some
high
availability
tools
or
potentially
you
know,
rely
on
some
providers
or
something
to
give
you
that
control
plane.
But
it's
completely
separate
from
the
clusters
right.
A
A
So
it's
kind
of
yeah,
it's
it's
essentially
a
it's
its
control
plane
only
right,
there's,
no
worker
nodes
or
anything
in
that
control,
plane
right
all
it's
doing
its
storing
desired
state.
It
doesn't
run
anything
on
its
own.
It
just
controls
the
other
clusters.
So
it's
pretty
lightweight
from
that
aspect.
A
Yeah
so
great
question
right
now,
we're
kind
of
so
yes
they're
first-class
objects.
We're
currently
borrowing
the
cluster
spec
definition
that
the
Federation
came
up
with,
which
is
not
really
sort
of
a
resource
first
class,
because
it
only
describes
the
clusters
that
were
pre-built,
but
we
kind
of
flipped
it.
So
we
are
using
the
cluster
spec
as
a
source
to
manage
the
clusters.
There
is
a
new
working
group
right
now
is
trying
to
declare
cluster
api's
and
new
specs
about
managing
clusters
and
whatnot.
A
But
your
second
part
of
the
question,
if
I
understood
it
correctly,
is
that
the
complexity
of
kind
of
making
some
policy
based
decision
about
which
apps
needs
to
run
on
what
clusters,
even
if
it
has
to
build
the
clusters?
Yes
I
mean
there's
a
feature
in
Federation
called
cluster
selector.
It
will
give
you
some
basic
sort
of
segmentation,
but
obviously
the
segmentation
is
not
at
this
point,
limited
to
what's
actually
running.
If
the
segmentation
chooses
to
start
using
some
clusters
that
are
kind
of
idle
or
offline,
it'll
start
bringing
them
up
and
whatever.
A
So
you
do
have
some
capabilities
to
start
sort
of
like
label
based
selector
type
of
decisions.
So
you
can,
you
can
have
a
bunch
of
clusters
with
your
label
production
or
QA
or
west
or
east,
or
you
can
come
up
with
whatever
labels
you
want.
You
can
have
your
developers
bring
their
constraints
basically
saying
I
want
an
app
I
wanted
to
run
in
Europe
I
want
it
to
be
secure,
I
want
it
to
be
whatever,
but
they're,
never
picking
clusters,
all
they're
doing
is
they're
giving
you
conditions
that
need
to
be
satisfied.
A
It's
your
responsibility
when
or
whatever
on
the
navarchus
aside,
that's
kind
of
the
whole
point
of
this
thing.
Really
it's
the
decouple
that
developer
flow.
If
you
will
from
the
actual
infrastructure
in
a
way
of
you
know,
you
have
a
query
and
you
need
to
satisfy
that
query.
Yep
all
right,
that's
it!
Thank
you.