►
From YouTube: Kubernetes SIG On-Prem Meeting 20170301
Description
Agenda/minutes:
https://docs.google.com/document/d/1AHF1a8ni7iMOpUgDMcPKrLQCML5EMZUAwP4rro3P6sk/edit#heading=h.nrh4k3ck5icu
Mailing list:
https://groups.google.com/forum/#!forum/kubernetes-sig-on-prem
Note from the meeting:
Agenda:
Demo/Presentation: K8s via Kargo using Digital Rebar on Hybrid by Rob Hirschfeld (RackN) & Greg Althaus (RackN)
## Demo:
https://github.com/digitalrebar/digitalrebar
OSS project called Digital Rebar: http://rebar.digital
Deploying kubernetes via Kargo; on-prem and cloud
Bare metal, bios discovery, lay down OS, etc. send workloads which are kubernetes deployments
Digital Rebar is a bunch of containers all managed together;
https://rackn.files.wordpress.com/2016/02/hwmaas.jpg
## Federation state on premise (mattymo)
What is current state?
https://github.com/kubernetes/kubernetes/issues/40536
What gaps need to be closed to get decent UX?
https://github.com/kubernetes/kubernetes.github.io/issues/2197
https://github.com/kubernetes/kubernetes/issues/39271
Second Agenda Item:
Federation state on premise
Are members aware of federtion on prem? Requires external IP controlling for controlling and connect to an outside DNS.
Kubefed needs to be updated to support but it’s not in the syntax, but there is provider there.
Should we use NodePort, Mirantis ExternalIP for deploying controlling the federation API server.
How are you solving this problem now?
Most are not running federation on-prem?
Toy playing using Kargo to do cross-cluster stuff, but with regard to networking this is abysmal and indicates a “don’t do that.”
The most likely use case for private/public. Local bare metal and once you reach limit you do on-demand on your cloud-vendor and use cloud to do up-down scaling. You don’t want latency between on-prem and cloud and with cni you should be able to do ingress between clusters. It just doesn’t work yet unless you point both on-prem and cloud to Google.
Group to discuss more next week
A
Alright,
now
that
we
hit
the
record
button,
I'm
going
to
go
ahead
and
start
over.
I'm
jorge
castro,
I'm
going
to
be
your
host
today
for
sig
on
prem.
We
have
two
agenda
items:
dumbass
consultation,
cooper,
dead,
easy
via
cargo,
using
digital
rebar
on
a
hybrid
cloud
with
rob,
hirschfeld
and
greg
alpheus.
I
hope
I
pronounced
that
right.
They'll
have
you
guys,
introduce
yourselves
in
a
minute
and
then
we're
going
to
talk
about
the
state
of
Federation
on-premise?
A
A
C
Okay,
great
I'll
telephone
gpo
reckon
we
Shepherd
a
open
source
project
called
diesel
rebar,
and
that's
what
we're
going
to
talk
about
today
with
regard
to
using
it
to
deploy
cooper
Nettie's
through
cargo
and
on
on
prem,
but
we
also
can
do
cloud
instances
as
well.
So
let
me
see
if
I
can
share
my
screen.
Give
this
a
show.
C
It's
a
basically
a
collection
of
deployments
on
a
set
of
nodes,
and
so,
in
this
case,
I
have
to
coover
Nettie's
deployments
in
a
system
deployment,
this
employment
or
we
use
to
discover
systems
and
then
the
other
two
ones
running
a
bunch
of
kvm
instances,
kind
of
accumulating
hardware
and
the
other
is
a
GCE
cloud
running
cooper
Nettie's
as
well.
So
one
of
the
things
we've
done
is
we
provide
a
digital.
Rebar
itself
provides
a
provisioning
environment.
C
We
can
do
bare
metal
hardware
discovery,
we
do
bias
configurations
raid
configuration
then
lay
down
os's
and
then
provide
that
to
the
rest
of
the
system
as
those
that
then
can
be
consumed
to
deploy
a
workload,
and
in
this
case
the
workloads
are
these
deployments
so,
like
I,
have
running
on
a
kvm
instance,
a
my
Cooper
Nettie's
deployment,
oh
well,
assuming
I.
Have
it
not
quite
set
up
right
because
it's
running
on
my
desktop
at
home,
but
in
my
cloud
instance,
go
back.
C
C
The
idea
is
that
we're
using
the
upstream
playbook
directly,
the
ansible
based
digital
rebar,
allows
you
to
sequence
and
control
sorts
of
open
source
or
your
own
private
kind
of
what
we
call
roles
in
this
case.
We're
running
the
ansible
playbook
directly
from
cargo
there's
other
things.
We've
been
playing
with
things
like
OpenStack
deployed
through
helm
on
top
of
coover
Nettie's
from
me.
There's
a
group
in
fact
I
think
it
well.
It
was
in
the
slack
for
crew,
benetti's
I
think
they
just
got
accepted
into
the
hope.
C
C
Hey
I'm
going
to
start
another
kvm
instance
and
what
digital
rebar
provides
for
an
on-premise
kind
of
setup.
Is
the
ability
to
dynamically
discover
hardware?
Add
it
to
the
system,
discover
it
our
case,
inventory
it
and
then,
as
the
system
comes
along,
add
them
to
the
to
an
existing
environment.
We,
like
I,
said
we
mentioned.
We
take
advantage
of
cargo,
so
we
have
node,
and
cargo
has
a
set
of
features
that
allow
it
to
do
adding
nodes.
It
can
also
upgrade
nodes.
C
We're
in
the
process
of
those
are
new
features
that
have
recently
been
added
and
so
we're
working
to
add
those
into
work
flows
as
well.
So
anyway,
this
is
my
KTM
instance
booting,
powerful
reviews.
We
build
up
an
inventory
of
what
that
system
is,
in
this
case
infant
Akkadian
instance.
We
won't
necessarily
see
as
much
in
a
way
of
like
I.
Can
my
infrastructure
stuff
like
that,
as
the
system
goes
through
its
discovery
process,
digital
rebar
will
assign
IP
addresses.
These
can
be
admin
interfaces
we
do
before
and
b6
addressing.
We
also
do
out-of-band
management.
C
So
if
an
ipmi
controller
has
been
detected
that
can
be
configured
and
enabled
as
well
with
those
set
up
and
configure
the
system
can
then
be
driven
through
the
rest
of
the
process.
What
happens?
Is
the
system
pixie
boots
by
default,
or
we
expect
systems
to
pixie
boot?
We
give
it
an
image
called
sledgehammer,
that's
basically
a
simplest
of
and
RAM
only
image
it
goes
through
and
does
the
discovery
process
at
this
point
the
node
could
have
raid
controllers
configure
discovered
could
have
BIOS
detective
that
needs
to
be
set.
C
The
node
into
my
kvm
deployment-
and
it
will
show
up
in
here-
and
at
this
point
I
can
say
I
want
to
edit
the
deployment-
and
I
can
bind
things
to
it.
So
at
this
point
I
can
say
I
want
it
to
become
a
cur
benetti's
worker.
So
I'll
add
it
to
that,
and
the
role
sets
up
handling
the
dependency.
So,
for
example,
creating
worker
knows
that
it
needs
to
have
an
LS
installed.
So
that
will
then
add
that
at
this
point,
I
can
commit
it.
C
The
committing
should
cause
it
to
take
effect
or
will
cause
the
tick
effect.
At
this
point,
the
kvm
instance
will
reboot
go
through
an
install
cycle
installed
base
OS.
In
this
case
the
OS
is
1604,
though
it
could
be
all
sorts
of
other
platforms.
The
environment
allows
us
to
keep
track
of
what
we
call
boot
environments
to
make
things
available.
Those
can
be
things
like
local
environments
are
sledgehammer
discovery
image.
We
have
it
very.
Some
stall
options
are
available.
C
The
black
ones
indicate
that
the
base
image
isn't
available
in
this
environment
right
now,
so
it
could
be
added.
This
also
can
be
extended
to
include
things
like
hardware,
validation,
images
and
things
like
that
as
well,
and
so
you,
basically
normally
what
you
would
expect.
You
have
normal
boot
parameters,
templates
that
need
to
be
built
for
each
node,
and
then
we
have
a
templating
system
that
lets
you
construct
whatever
you
want
for
your
boot
environment
right.
C
C
C
C
Then
we
can
choose
layout
whether
we
want
to
be
masters
or
workers
and
choose
a
che
or
not.
So
I
can
choose
an
h,
a
deployment
this
case
encore
both
but
we'll
set
that
running.
We
get
a
JSON
blob
that
represents
this
deployment
so
that
we
could
save
it,
reuse,
it
inject
it
through
the
fuel,
I
and
then
I
can
set
this
off
and
running,
and
it
will
go
out
to
my
amazon
account,
which
is
provided
through
the
provider.
C
C
So,
with
regard
to
networking
within
the
providers,
we
in
general
take
the
default
view,
pcs
that
are
configured
in
the
providers
for
bare
metal
it's
much
richer.
We
have
the
concepts
of
being
able
to
define
multiple
network,
so,
for
example,
this
is
the
admin
network
that
we're
running
the
kvm
instances
on
right.
Now,
it's
just
a
bevy,
it's
on
an
untagged,
unbridged,
unbonded
environment,
and
then
you
can
define
a
host
address
space
which
is
kind
of
static
in
a
DHCP
ranges
to
hand
out
those
are
all
options
for
the
admin
network.
C
Additionally,
you
can
add
follow-on
networks
as
well,
so
that
you
can
define
like
I've
got
us.
I
need
to
create
a
bonded
pair
with
VLANs
that
need
to
be
exposed
to
separate
networks.
Those
can
then
be
exposed
and
constructed,
additionally,
will
do
things
like
grouping
them
so
that
we
know
that
if
it
came
from
this
admin
network,
it
should
probably
pull
from
this
group
of
public
addresses
as
well.
Those
kind
of
concepts
are
all
handled
through
the
network
administration
part
it's
fairly
rich.
C
We
have
a
conduit
abstraction
layer,
so
the
idea
is
that,
instead
of
saying
you
want
each
0,
we
actually
say
you
want
like
10
g
1,
which
would
be
the
second
10
gig
interface.
That's
detected
on
the
system.
That
way
you
can
handle
bus
enumeration
problems
as
well
as
OS
enumeration
right.
Some
OSS
call
them
each
0,
some
call
them
emt,
5s
34
right.
Second,
so
that's
the
intent
to
the
conduit
obstruction.
C
B
C
There's
various
views
on,
for
example,
in
a
bare
metal
space,
the
what's
the
in
one
space,
which
is
good
by
some
of
the
attributes.
We
gather
as
part
of
the
discovery
image
are
things
like
lldp
information.
We
try
and
gather
port
objects,
which
I
can't
show
you
to
be
you
I
right
now,
but
the
idea
and
kvm,
so
there
wouldn't
be
any
data
anyway.
The
idea
is
that
we
gather
ldp
information
so
that
you
can
get
a
view
of
what
switch
ports.
C
Things
are
connected
to
there's
been
long
term
dreams
for
racking
to
add
to
digital
rebar,
the
ability
to
manage
switches,
but
that's
not
been
a
focus.
We
find
that
within
the
organization's
a
lot
of
times.
These
are
two
separate
groups
that
aren't
necessarily
communicating
great.
So
it's
not
been
a
high
priority
item
for
us,
but
we
have
the
data
to
be
able
to
suggest
things
like
LCP,
bonds
and
stuff
like
that
now.
One
of
the
other
aspects
that
we
try
and
deal
with
with
regard
to
deployment
is
the
idea
that
you
may
wish
to
redeploy.
C
We
need
to
re,
deploy
nodes
so
part
of
what
we
handle
is
life
cycle.
So
we
have
the
ability
to
say
rebuild
this
whole
system,
so
force
them
through
a
reinstall
or
in
the
case
of
GCE
reapply
the
systems
or
you
can
do
it
on
a
perdon
node
basis.
So
the
idea
is
that
you
can
force
a
note
to
reinstall
itself
back
to
what
it
was.
Obviously
that's
dependent
upon
how
you
chose
to
set
up
a
che
and
things
like
that.
C
D
C
Integrate
this
is
a
rebar
also
has
behind
it
self
an
event
system
that
it
can
set
up
them,
send
events
to
itself
internally.
A
lot
of
these
things
are
handled
through
events,
but
you
can
also
set
up
listeners
for
them.
One
of
the
ways
we
have
to
do
that
is
through
what
we
call
our
rule
engine,
and
that
has
the
ability
to
specify
things
like.
C
D
C
The
discovery
from
we
run
a
wealth
of
money.
Two
answers
to
that
one
is
our
goal
is
for
have
for
physical
infrastructure.
To
always
picks
you
do
the
DHCP
server
that
we
run.
We
can
use
your
existing
one,
but
we
also
provide
one
as
part
of
digital
reboiler
that
tells
the
nodes
to
boot
sledgehammer
on
discovery
and
then
keeps
track
of
it.
What
it's
dude
environment
should
be
going
forward
and
eventually
that
becomes
a
local
and
then
drag
do
from
local
disks
right,
and
so
that's
how
that's
being
managed
does
that
answer
your
question
yeah.
C
C
What
we
expect
on
the
delegation
of
dhcp
server,
so,
if
you're
actually
wanting
to
run
your
own
network
infrastructure
and
network
I
pan
and
all
that
kind
of
stuff,
then
what
we
ask
is
for
those
in
bio
for
those
nodes
that
you're
wanting
digital
rebar
to
manage.
You
would
point
the
next
boot
server
to
our
our
system,
to
the
provisioner
part
of
our
system
and
I'll
show
you
that
real
quick
to
give
you
an
idea
of
how
digital
rebar
is
actually
working.
It
just
works
I
think.
B
C
C
Then
we
have
our
kind
of
core,
dr
services,
which
are
reverse
proxy,
providing
kind
of
single
sign-on
access
and
then
an
API
engine
and
database
and
service
registry
and
logging
system,
and
then
we
also
have
our
cloud
wrap,
which
is
what's
managing
our
outbound
API
to
like
manage
Google
and
Amazon
and
open
second
packet.
Google
engine
does
that
we
run
our
own
chef
server.
Some
of
our
roles
are
provided
by
chef,
so
we
can
inject
your
own
cookbooks.
C
We
could
also
make
that
external
as
well,
and
then
we
run
a
pki
service
to
handle
the
ability
to
generate
signed
certificates
within
the
infrastructure,
but
it's
also
exposed
so
that
it
can
be
used
to
create
certificates
for
workloads
if
they
wanted
as
well
and
they
create
separate
root
domains
or
this
the
environment.
So.
C
So
what
happens
is
as
part
of
the
workflow,
the
some
when
it's
discovered
shows
up
and
get
these
basic
roles,
and
at
this
point
it's
sitting
in
what
is
called
the
node
is
marked
with
a
boot
environment.
So
if
I
go
look
at
this
node,
it's
now
in
local,
so
what
happens
is
on
big
gut
where
you
put
into
sledgehammer
okay,.
D
C
So
what
happens
is
the
discovery
image.
Is
that
once
you
apply
a
set
of
rules
specifically
the
rebar
installed
node
roll,
it
will
change
the
good
environment
to
whatever
the
install
images
I
think
goes
through
installs
the
image
when
that
complete
the
last
step
of
the
kickstart
file,
that's
it
to
lose
local.
D
C
So
that's
kind
of
how
the
backend
provisioning
pieces
are
once
an
OS
is
installed
and
ssh
keys
are
in
place.
The
system
then
kind
of
normalizes
itself
to
kind
of
cloud-based
operations
or
style,
where
it's
been
kind
of
doing
all
the
Restless
stuff,
through
your
like
ansible
or
chef
it'll,
just
whatever
and
that's
applicable
across
both
like
the
cloud
and
shows,
as
well
as
the
physical.
A
C
As
we
come,
it's
actually
configurable
through
metadata.
So
right
now
we
do
a
kind
of
an
intermediate
repo
which
is
basically
just
exactly
there's
no
difference
between
them
other
than
just
to
represent
a
stabilization
point.
Car
goes
in
the
process
of
the
finding.
What's
their
release,
cadence
infinity
and
so
we'll
probably
reference
the
release
Caden
more
seconds
more
set.
The
goal
is
to
use
up
strings
were
possible
in
some
cases
like
for
the
OpenStack
helm
stuff
that
we've
been
playing
with.
We
have
a
repo
that
has
one
more
fix
in
it.
D
So
so
the
point
I
would
make
is
there's
two
things:
what
is
we're
really
are
just
using
the
ansible
right?
We
drive
an
to
the
bullets,
it's
pretty
straightforward
that
perspective.
We
want
to
be
using
cargo,
so
we're
not
we're
not
because
you
get
the
benefit
of
community
because
subscrib
you
back
to
the
community.
You
could,
and
we
talked
about
this
at
a
cluster
ops
meeting.
You
know
you
could
use
what
we're
doing
and
stage
cops.
D
If
people
wanted,
you
could
stage
crew
bad
men,
because
the
work
we're
doing
or
digital
rebar
is
really
a
ready
stating
platform.
So
it's
designed
to
get
infrastructure
up
into
a
certain
set
and
create
a
deployment
which
is
a
so
foundry
and
take
actions
against
that
set
of
machine
and
then
use
tools
that
already
exists
right.
We
don't
want
to
create
unique
deployment.
Stuff
right.
These
we
could
drive,
do
too
I'll
fun
with
this
come.
D
This
is
on
the
stage
and
then
launch
and
then
once
it's
done
we're
aware
of
it,
then
we
can
actually
take
acting.
The
OpenStack
stuff
actually
drive
home
and
then
go
feel
some
special
point.
On
top
of
the
Cuban
Eddie
suppose
you
know
when
the
choice
finish,
so
you
can
chain
different
operational
models
together
and
mix
master.
Oh
great,
we're
going
to
show
where
the
yeah.
C
So
our
inventory
is
built
based
upon
metadata,
and
so
in
some
regards
it
can
be
empty.
It
can
be
full
and
some
regards
we
have
two
levels
to
deal
with
that
in
our
inventory.
This
section
here
of
our
definition
for
like
this,
is
defining
a
role
in
in
this
mu
and
in
fact
this
one
is
the
crew,
benetti's
master
role,
for
example,
and
so
right
now
what
we're
doing
and
we're
going
to
switch
this
to
even
use
more
of
the
upstream
part.
The
point
is:
there's
a
lot
of
flexibility.
C
So
if
you
wanted
to
build
your
own
answerable
playbook
and
run
them
against
normal
or
just
your
deployed
systems,
you
can
do
it.
In
this
case,
we
use
some
maps
to
say
what
should
things
become
so
this
group
map
here
takes
digital
rebar
rolls.
So
if
it's
a
case
worker,
then
it's
going
to
get
these
inventory
group
inside
of
the
inventory
file.
C
That's
how
that
gets
red
and
then-
and
we
also-
which
we
right
now
are
running
each
roll
independently
in
cargo,
and
that
was
an
attempt
to
allow
you
to
get
more
visibility
into
the
sequencing
of
what
was
going
on,
but
I
think
we're
going
to
start
changing
that
to
be
more
more
conducive
to
upstream
and
just
use.
The
cluster
files
provided
by
the
upstream
and
so
right
now
what's
happening
is
we're
saying
on
things.
C
They're
listed
case
master
run
this
set
of
roles
from
within
the
inside
of
the
ansible
playbook,
and
then
we're
going
to
inject
attributes.
So,
for
example,
these
are
global
attributes
that
get
injected.
So
we're
going
to
inject
the
HTTP
proxy
and
other
stuff,
and
then
we
even
do
per
node
injection
so
that
we
can
do
things
like
the
access.
Ip
should
be
the
thing
listed
as
the
Kate
master
network.
Take
the
private
address
and
use
that
for
that
value
inject
for
the
node
and
IP.
But
this
is
the
private.
C
So,
in
this
case,
we're
forcing
it
to
be
all
internal
to
look
internal,
but
you
could
tweak
these
such
as
you
could
do
things
like
use
cargo
to
deploy
nodes
across
Amazon
and
Google
by
providing
the
public
ip's
as
the
access,
I,
peas
and
the
internal
IPS
to
be
the
private
eye
piece,
you
could
tweak
those
kind
of
things
with
you.
Okay,.
E
C
C
So
if
I
go
look
at,
for
example,
one
of
the
deploy
files
I
can
look
at
Kate
config,
for
example,
and
this
is
all
of
the
attributes
that
are
currently
a
subset
of
the
attributes
there
to
put
exposed
that
could
be
tweaked
right.
Some
of
them.
You
don't
want
to
tweak
some
of
the
meetings,
but
you
could
change
these
and
then
it
would
rerun
all
the
playbooks
that
were
associated
it
be
variable.
C
So
now
we're
so
you
can.
You
can
do
an
unsafe
upgrade,
for
example,
right
now,
just
by
going
and
changing
dr.
12
to
dr.
one.
You
know
one
about
13
right
and
that
will
go
forcibly
reset
it
right.
I
know
in
the
case
of
cargo
they've
added
an
additional
playbook
file,
cluster
file
that
handles
upgrade
more
gracefully,
and
so
one
of
the
things
that
there's
on
my
to-do
list
is
to
add
a
deploy
and
an
upgrade
that
calls
both
of
those
based
upon
what
they
do.
C
D
And
and
these
attributes
are
soaked
based
on
a
deployment,
so
you
can
have
one
deployment
with
one
set
of
attributes
and
then
the
next
next
appointment
with
the
next
set
of
attributes
and
they
can
come
from
multiple
Dickens
they
cascade
downwards.
So
if
you
have
attributes
from
the
hardware,
that's
discovered
or
you
want
to
set
it,
then
you
can
stay
to
the
stack
things
together
or
retrieve
information
from
a
deployment.
After
it's
done
it
becomes
an
attribute.
Then
you
know
what
depends
on
what.
So
one
of
the
things
that
stress
is
really
important.
D
Is
that
when
you
take
an
action
from
a
role,
if
the
role
generates
data,
you
have
a
way
to
create
baby,
build
attributes
from
actual
data,
discovered
information
and
then
feedback
downstream,
and
then,
if
it
changes
its
going
to,
if
it's
a
trigger
a
cascade
of.
Oh,
you
needed
this
information
that
information
change
and
you
know
who's
consuming
it,
because
the
system's
actually
has
a
graph
and
it
tightly.
The
attributes
are
tightly
coupled.
C
When
the
other,
the
other
thing
we
provide
is
each
role
within
this
rebar
has
its
own
log
log
tracking.
So
you
can
see
what
goes
wrong
and
since
deployment
is
built
as
a
as
a
graph,
the
idea
is
that
the
graph
is
sequence,
but
if
something
has
an
error
that
will
turn
red
the
process
will
pods
for
that
deployment.
For
that
node
and
remediation
can
be
done
and
then,
as
appropriate.
The
role
can
be
retried
and
process
can
read,
is
to
fail
fast
and
give
feedback,
but.
C
C
A
If
you
guys
would
just
go
back
through
the
notes
and
make
sure
that
you
have
URLs
for
the
stuff
that
you
want
to,
maybe
I'm
weird
experimental,
putting
the
note
to
my
show
youtube
video
description
so
that
we
can
find
things
months
from
now.
So,
if
there's
anything
that
you
want
to
correct
or
expand
acronyms
or
any
of
that
stuff,
I'm.
C
A
A
E
That's
that's
me.
Hopefully,
there's
no
people
shouting
in
the
background
right
now.
Those
are
our
so
I
was
wondering
if
any
of
the
members
present
are
aware
of
the
state
of
Federation
on
promise.
Well,
the
main
problem
is:
it
requires
a
cloud
provider
to
work
because
it
relies
on
external
IP
controlling
for
the
Federation
API
server
first
was
from
the
next
tunnel
IP
and
then
to
connect
to
a
Guinness,
a
cloudy
and
a
service
either
Google,
GCE,
gke
or
AWS.
E
E
Stock
cube
Fed
needs
to
be
updated
to
support
it,
but
it's
not
actually
in
the
syntax
anywhere
I
get,
but
there
is
a
primer
there
and
then
the
other
option
issue
is:
should
we
use
node
port?
Should
we
consider
parenthesis
extra
likey
provider,
controller
side
or
some
other
options
for
deploying
controlling
the
Federation
API
server.
E
F
B
F
C
We've
been
mostly
waiting
to
see
what
shows
up
with
a
guard
election
of
the
uber
Nettie's
work,
but
not
sure
we
did
some
toy
playing
with
things
like
using
cargo,
for
example,
or
the
when
we
were
using
stuff
before
cargo
to
do
cross
cluster
stuff,
but
in
some
regards
with
regard
to
networking,
that's
just
kind
of
abysmal
and
we're
not
sure
why
anybody
would
necessarily
want
to
do
that
just
other
than
just
to
show
it
could
be
done.
Oh
you
shouldn't.
E
E
D
E
Let
me
illustrate
the
most
prime
example
of
why
you
would
want
public
private
Federation
one.
Is
you
have
your
your
your
physical
servers
you
paid
for
all
of
them
so
use
deploy
as
much
as
you
can
all
the
resources
you
can
locally
and
then
once
you've
reached
your
limit,
you
start
provisioning,
pods
and
whatever
on
your
your
cloud
vendor
on
demand
and
those
costs
money,
you
know
when
you're
using
them
and
then,
when
your
demand
drops
you,
you
know,
get
rid
of
them.
Eventually,
that's
much
more
difficult,
but
you
can
still
do
it.
E
So
you
want
them
separate,
because
you
don't
want
a
very
high
latency
between
your
cubits
and
your
API
servers,
because
and
especially
your
your
CNI
there's,
there's
no
need
for
them
to
be
close.
If
you
have,
you
know
how
to
say
some
sort
of
routing
an
external
IP
controlling,
so
you
can
do
ingress
between
the
two
clusters.
F
Yes,
a
look,
look
like
it's
a
quite
a
problem
now
right,
so
so,
I
think
that
might
be
a
good
candidate
to
start
working
on
actually
from
from
Tegan
front
perspective.
So
looks
like
we
have.
Probably
we
need
some
more
time
to
think
about
it,
so
maybe
for
the
for
the
next
meeting
will
travel
to
gather
some
some
existing
solutions.
Solutions
out
there,
maybe
talk
to
Federation,
guys,
will
add
their
dead
plants
and
we
can
figure
out
how
we
can
have
them.
Actually,
with
this.