►
Description
Cameron's Topic Summary:
Kubernetes is taking the market by storm with remarkable levels of ecosystem and enterprise adoption. Not to mention one of the most robust communities of contributors in the open source world. While Kubernetes is fabulous for deploying, managing and scaling microservice-based applications, the enterprise is a brown field of technologies and workloads that don’t always fit neatly into the microservice mold. Navops Command augments Kubernetes by allowing organizations to not only share resources in Kubernetes clusters across projects and teams, but allows for the running of non-containerized batch workloads in the same environment, thus addressing some of the more complex, legacy needs of the enterprise. This talk will discuss resource management within Kubernetes and will include a demonstration of running complex, mixed workloads in a shared environment.
A
Okay,
fantastic,
and
thank
you
very
much
for
that
introduction
and
having
us
here.
You
know
invited
to
your.
Your
media
were
very
much
looking
forward
to
it
so
to
get
started
first.
This
is
going
to
be
a
presentation
regarding
premature
Bernays
resource
management,
our
views
into
that,
as
well
as
some
products
that
we've
created
to
kind
of
address,
some
of
the
things
that
we
think
could
really
be
improved
in
kubernetes
or
in
kubernetes
environments,
where
you
have
mixed
workloads
or
resource
constraints.
A
Our
release
was
last
November
that
we
feel
brings
a
lot
of
our
ideas
to
the
kubernetes
space
in
regards
to
scheduling
a
new
workload
management,
but
if,
in
the
command
section
we'll
talk
about,
obviously
the
product
also
talk
about
our
thoughts
on
mixed
workload,
solutions
and
show
a
demo
of
both
command
E
and
some
mixed
workloads
in
action.
Finally,
we'll
wrap
it
up
with
some
Q&A.
A
So
who
are
we
so?
As
I
said,
I'm
Tamara,
Brunner
chief
architect,
I've,
been
here
for
about
ten
years
at
universe
of
different
things,
starting
at
software
software
development
and
then
moving
up
here
into
the
more
architectural
role.
I've
worked
on
pretty
much
all
of
our
products.
At
this
point,
and
specifically
the
ones
of
note
here,
our
grid
engine,
since
that
will
be
that's
a
core
part
of
our
native
apps
chameleon
product.
A
Additionally,
I
have
Stefan
Hafiz
a
new
remember
on
the
nav
ops
team,
but
he
has
been
here
at
univille
a
long
time
core
developer
on.
You
have
a
grid
engine
and
our
chief
designer
and
technical
lead
on
our
license:
Orchestrator
product,
which
is
a
license
management
solution
for
managing
proprietary
licensing.
In
a
kind
of
a
high
performance,
compute
cluster
and
then
who's
Unova,
so
what
we?
What
we
are
as
a
software
company
that
produces
solutions
for
workload,
orchestration
and
container
management?
Typically
in
large-scale
compute
environments,
our
customers
are
generally
large
enterprises.
A
You
can
see
there's
a
sample
there
at
the
bottom,
in
some
of
the
sectors
where
we
have
have
some
customers,
life
sciences
and
energy
are
two
very
large
segments
for
us.
We
have
offices
throughout
the
world
I'm
in
an
engineering
office
here
in
Chicago,
as
Stefan's
actually
in
the
engineering
office
they're
in
our
Regensburg
Germany,
and
we
also
where
the
home
of
Grid
Engine.
So,
if
you've
ever
heard
throughout
your
career,
the
term
grid
engine,
maybe
working
with
an
HPC
in
an
HPC
context,
we're
pretty
much
producing.
A
We
have
an
official
home
for
that
product,
producing
a
commercial
version.
We
also
released
the
most
open
source
versions.
At
this
point,
we
employ
a
lot
of
the
folks
who
wrote
the
open
source
versions
over
the
last
several
years.
Just
for
Stefan
came
from
as
well,
as
you
know,
taking
kind
of
a
transfer
of
ownership
from
Oracle
after
they
had
bought
son
and
acquire
grid
engine
so
now
I'm
to
scheduling
and
workload
management
and-
and
really
you
know
from
our
background
in
the
HPC
space,
we've
managed,
you
know
thousands
of
applications
on.
A
You
know
hundreds
of
customers
over
about
20
years.
That's
how
long
our
team's
been
working
in
in
this
space.
We've
learned
a
lot
of
few
things,
a
lot
of
things
about
them
about
scheduling
and
workload
placement
and
what's
kind
of
consistently
important
across
all
these
environments.
So
one
thing
that
obviously
becomes
very
apparent:
is
we
live
in
a
world
where
resources
are
finite?
Obviously,
that's
easy
to
see.
A
If
you
walk
to
your
lab-
and
you
see
you
know,
you
have
five
machines
and
you
can
fill
them
up,
but
this
extends
really
to
any
type
of
platform.
I
mean
typically
you'll.
Have
you
know
budgets
placed
on
you
when
using
cloud
resources,
you
could
have
quotas
places
on
you
as
well,
which
in
fluid
you
know,
enforce
a
limit
or
resource
constraint
on
the
entity
being
limited
by
the
quota?
So
it's
not
infinite.
A
We
need
to
consider
that
when
scheduling
work,
the
another
attendant,
of
course,
is
that
people
really
want
to
get
as
much
done
as
possible
as
as
possible
with
resources
that
they
have.
So
you
know
if
they're
investing
in
in
local
hardware
or
if
they're,
if
they
have
their
quota,
they
want
to
make
sure
that
they
can
be
as
productive
as
possible.
So
this,
of
course
needs
means.
You
need
to
meet
application.
A
Sla
is,
if
you're
writing
in
your
cluster,
something
that
needs
to
be
up
all
the
time
that
needs
to
continue
to
be
off,
but
you
also
need
to
be
able
to
use
your
resources
to
make
sure
you
can
move
that
product
forward.
Do
effective
testing
of
that
product
and
be
able
to
get
information
about
your
processes
in
terms
of
improving,
improving
the
process
of
improving
efficiency,
to
be
able
to
get
more
work
done,
be
able
to
get
your
products
released
faster
and
provide
better
uptime.
A
So
you
may
ask
what
some
what's
required
of
a
scheduler
that
needs
to
be
pretty
much
the
manager
of
all
the
workload
across.
You
know
several
groups
or
in
heterogeneous
workloads,
and
you
know
multiple
teams
and
groups,
and
we
really
believe
these
three.
These
three
points
here
are
some
of
the
primitive
important
ones.
So,
specifically,
a
prioritization
is
important.
As
I
described
earlier,
you
can
run
into
resource
constraints,
you'll
run
into
it.
You
know
with
budgets
or
the
quotas.
Well,
you
know
with
prioritization.
A
You
can
then
determine
you
know
not
only
which
resource
gets
to
which
job
gets
to
run.
Next,
when
you
have
run
up
against
one
of
those
limits,
you
also
allow
higher
or
more
important
workloads
to
get
access
to
resources
that
are
more
favorable.
You
know,
often
a
workload
could
run
on
something
less,
but
it
run
better
if
it
had,
you
know
some
more
RAM
or
some
more
CPU
available.
A
This
will
allow,
you
know,
allow
you
to
have
your
more
important
resources
get
what
they
need
and
then
allow
the
lower
the
lesson
for
resources
to
get
placed
on
you
know
minimal
or
whatever
is
remaining
also
more
information
for
the
schedule
or
more
information.
That's
useful
that
the
scheduler
has
the
better,
knowing
obviously
knowing
more
about
the
resources
that
are
available
in
the
environment,
knowing
what
type
of
work
wants
to
run
when
it
wants
to
run.
When
does
it
need
to
complete
really
all
in
the
end?
A
Finally,
you
need
to
have
a
method
to
generate
information
regarding
the
scheduling
decisions
and
then
be
able
to
analyze
the
scheduling
decisions,
to
make
sure
that
your
policies
are
being
met
and
that
the
business
seats
are
being
met.
This
can
allow
you
to
additionally
manage
for
future
capacity
growth.
If
you're,
you
know
you
need
in
six
months
to
be
able
to
double
your
compute
capacity.
A
Having
this
type
of
information
would
let
you
know
what
resources
you
need
to
buy
in
order
to
improve
your
food
by
by
it
may
not
be
buying
twice
as
many
machines,
for
example.
It
also
allows
you
to
identify
bottlenecks,
and
this
will
be
where,
if
you
have
quotas
or
machine
constraints
that
don't
really
match
the
workloads
that
you're
coming
in,
like
let's
say
you
only
have
10%
g2u
machines,
but
90
percent
of
your
mission.
Your
jobs
are
one
in
GPUs
you'll,
be
able
to
detect
that
and
see.
A
Wow
I
need
to
buy
some
more
machines
with
GPUs
in
order
to
get
better
throughput
and
time
to
market
on
my
product
development.
So
this
brings
us
to
our
product
DevOps
command,
which
really
it
takes.
What
we've
learned
in
the
HPC
space
or
we've
applied
all
those
principles
that
I've
just
described
and
it
brings
it
to
command
I
mean,
brings
a
tag
to
kubernetes,
so
it
allows
you
to
run.
A
Take
kind
of
the
unpriced
workloads
are
currently
present
in
kubernetes
to
prioritize
them,
match
them
against
all
sorts
of
additional
policy.
Attachable
objects
such
that
you
can
get
tighter
placement
be
able
to
really
share
a
cluster.
Let's
rank
very
many
diff
types
of
workloads
across
many
different
types
of
groups
and
users,
while
still
maintaining
a
very
high
utilization
rate
of
your
cluster.
A
So
what
is
really
nav?
Ops
command?
It's
a
multi
container
application.
We
designed
it
is
very,
very
container
friendly
forward,
so
everything's
done
pretty
much
in
the
container
and
it's
actually
self
hosted
inside
a
kubernetes.
So
the
three
containers
that
exist
inside
of
the
main
nav
ops,
command,
pod
or
one
for
managing
the
scheduling
decisions,
one
for
bridging
the
data
model
between
kubernetes
and
our
internal
scheduler,
and
then
one
for
storing
our
internal
configuration.
We
additionally
have
a
container
that
runs
at
our
UI,
our
web
user
interface,
that
runs
as
a
separate
pod
allowing
them.
A
You
know
an
admin
user
to
be
able
to
configure
the
policy
and
the
scheduler
dynamically.
In
the
end,
users
interact
with
command,
just
as
they
normally
would,
and
this
is
because
command
implements
the
third-party
scheduler
interface
provided
by
kubernetes.
So
really
what
we?
What
we
do
is
we
just
plug
in
and
we
take
using
the
third-party
scheduler
annotation.
We
take
pods
that
are
either
targeted
directly
for
a
command
when
a
command
is
running
as
a
non-default
scheduler
or
we
would
take
even
the
default
pod,
so
pods
without
a
annotation.
A
If
you
configure
command
to
run
as
your
default
scheduler,
additionally
properties
can
be
added
to
an
a
Vox
I'm.
Sorry
kubernetes
pods,
to
leverage
more
of
the
features
inside
a
command,
but
this
is
all
done
through
standard
labels
and
annotations
inside
of
kubernetes.
So
really,
as
I
said
at
the
end
end
users,
they
don't
need
to
know
anything
about
command
in
order
to
use
it,
but
they
may
add
additional
attributes
and
annotations
in
order
to
leverage
certain
policy
insides
command.
A
So
the
key
value
here
is
the
the
scheduling
application
that
we
have
inside
of
command.
It's
actually
a
containerized
version
of
our
Cora
HPC
scheduler.
So
you
get
all
of
the
functionality
and
features
that
are
baked
into
that.
So
here's
a
kind
of
graphical
view
of
what
I
was
just
describing.
So
you
can
see
right
there
kind
of
the
communication
channels
between
the
nav,
ops
command,
pod,
the
kubernetes
api
server
and
then
the
actors
in
our
environment.
A
So,
as
you
can
see,
the
admins
primarily
the
one
who
interacts
with
the
with
the
system
to
configure
the
policy
and
that
can
be
done
through
the
web,
UI
or
nav
up
CTL
command-line
utility,
and
then
the
end
user
uses
qu
control.
Like
always.
The
line
right
down
there
at
the
bottom
is
of
note
is
that's
how
we
actually
tell
kubernetes,
through
the
binding
of
a
pod,
to
a
node
to
actually
schedule
the
pod.
A
A
So
so
we
have
provided
here
with
command.
This
was
the
release
that
we
did
in
November,
where
you
have
now
this
ability
to
add
a
complex
policy
to
your
kubernetes
scheduling
decisions,
and
that's
great.
You
get
our
fifteen
years
of
experience,
doing
that
and
a
lot
of
great
great
features,
but
we
found
that
some
of
our
customers
want
to
run
run
in
a
mixed
load
workload
environment.
What
that
means
is
really
they
have.
A
You
know
some
containerized
applications
that
run
inside
of
kubernetes
just
fine,
but
they
also
have
a
lot
of
workloads
that
don't
necessarily
run
under
kubernetes,
and
they
want
to
run
those
just
as
they
have
before,
without
really
modifying
anything.
So
what
we've
done
is
actually
fully
containerized
viewed
of
a
grid
engine
and
made
it
to
allowed
it
to
run
under
under
kubernetes
to
provide
pretty
much
an
HPC
cluster,
that's
hosted
by
kubernetes.
A
Now,
when
running
under
command
like
this,
do
you
additionally
have
command
placing
the
Unova
grid
engine
components,
a
unity
of
grid
engine
pods,
which
then
will
schedule
the
actual
HPC
work?
You
know
inside
of
them,
so
this
allows
you
to
run
both
grid
engine
in
a
dynamic
scaling
environment
on
top
of
kubernetes
and
be
able
to
run
your
legacy
unmodified
HPC
applications
with
that
being
said,
I'd
like
to
turn
the
presentation
over
to
my
colleague
Stefan,
where
he
will
describe
the
dev
environment
and
then
go
through
a
demo
of
this
functionality.
B
Hi
well
welcome
also
from
my
side
zone.
This
is
demo
environment
I
set
up
for
for
this
little
demonstration.
So,
let's
imagine
a
company
with
two
different
teams.
We
have
CI
CD,
so
continuous
integration
and
continuous
development
on
the
one
hand,
and
some
ops
team,
which
is
also
divided
in
a
batch
and
services
team.
We
have
a
couple
of
pots
and
workloads
running
later
on
in
the
demonstration
the
CI
CD
team
will
have
some
work
for
development
and
some
testing
work.
B
The
batch
team
has
the
DeGroot
engine
the
univariate
engine
running
inside
and
some
back
and
chopper
application
controller,
and
we
have
the
services
tasks,
on
the
other
hand,
which
does
administration
and
production
stuff.
This
is
running
on
so,
as
Cameron
said,
this
is
Carbonell
running
on
cue
Benitez,
which
has
never
command
on
top
of
it,
and
my
demonstration
is
simple
mini
cube.
We
are
so
I
need
to
take
or
share
my
screen.
No.
B
A
B
As
we
heard
already
from
Cameron
NAB
ups
command
provides
virtual
multi-tenancy
on
top
of
company
T's.
It
allows
to
share
one
cluster
among
multiple
teams,
as
I
said,
applications
and
services,
and
it
even
allows
to
run
containerized
and
non
containerized
workload
in
the
same
environment
by
running
UNIVAC
root
engine
as
the
workload
processing
service
for
these
non
containerized
applications
on
top
of
Khadijah's-
and
this
is
what
I
am
going
to
demonstrate
now,
let's
dive
directly
into
what
nabobs
command
mixed
workload,
support
allows
you
to
have
I'm
running
simian
ously.
B
So
you
have
you
see
here:
cute
control
get
pots
command
running
and
you
see
here.
We
have
running
currently
ugq
master
service
instance
and
we
have
running
to
ug
execution
daemon
instances
and
I
will
start
right
now,
a
couple
of
back-end
chips
and
front
and
chopped
controllers.
You
see
really
what
a
what
everything
is
about
that
I'm
you
I
will
say
later.
B
B
B
We
have
this
is
the
nabobs
command
and
web
UI.
On
the
left
side.
You
see
the
policies
you
can
manage.
Why
are
the
policy
management
they
I
will
show
you
how
to
change
things
like
the
proportional
shares
and
what
you
see
here
is
an
organizational
breakdown
of
how
the
entire
cluster
resources
are
to
be
used.
It's
reflected
in
the
proportional
share
policy
of
Napa's
commands.
The
full
width
of
the
diagram
represents
100%
of
the
cluster
resources
and
at
the
highest
level,
I
have
split,
and
these
into
between
CI
CD
and
the
apps.
B
These
are
the
teams.
I
have
shown
you
in
this
slide
below
CI
CD.
We
have
the
projects
for
development
and
test
for
simplicity
reasons.
I
will
not
run
any
part
within
those
projects,
otherwise
it
would
get
too
much
input
output
of
the
of
the
commands.
You
will
see
on
the
right
side.
We
have
the
services
team
and
the
patch
team
and
the
batch
team
is
also
divided
in
grid
engine
and
back-end
and,
as
you
can
see
here
is
some
some
kind
of
80/20
between
bread
back
and
and
great
engine.
Let
me
click
on
that.
B
For
that
to
see
that
a
bit
more
easy
I
used
an
additional
feature
of
nabobs
command,
the
so
called
runtime
quotas.
This
means
those
two
projects
are
limited
to
run
only
ten
parts
at
a
time.
So
we
should
see
later
that
there
are
running
aid
chopped
back
and
replication,
a
chop
back
and
chop
controllers
for
the
project
back-end
and
there
should
run
to
grid
engine
execution.
Human
hosts.
This
is
configured
as
I
said
here
in
the
runtime
quotas.
B
Nevels
command
automatically
creates
for
each
namespace.
You
have
in
your
cube,
Anita's,
cluster
and
entry
here.
For
for
that,
so
let
me
click
click
on
the
configure
quotas,
I
configured
already
a
limit
for
that
and
I
said
that
the
project's
grid,
entry
and
and
back-end
are
limited
to
two
to
run
ten
parts
at
a
time
yeah.
B
B
B
B
B
To
only
see
the
arranged
parts,
so
now
you
see
that
NAB
ups
command
scheduled
a
couple
of
another
execution
demons
and
stopped
a
couple
Nair
or
not
more
back-end
jobs
and
in
a
couple
of
seconds
we
will
see
here
on
this
screen
that
those
new
execution
demon
parts
will
automatically
get
added
to
our
univariate
engine
system.
And
yet,
if
there
are
pending
jobs,
which
are
waiting
for
the
resources,
they
will
get
automatically
used
by
the
great
engine
system.
So
a
third
one
God
already
added
to
the
cluster,
the
KH
k,
5q.
B
B
This
may
take
another
couple
second,
so
here
now
we
got
another
one,
so
this
cone.
So
now
you
know
what
what
have
you
seen
there
at
the
last
couple
of
seconds,
so
you
saw
a
single
queue:
bonitas
cluster
that
ran
standardized
standard,
containerized
workloads
along
with
univer
Grid
Engine
as
workload
processing
service
and
for
these
non
containerized
workload.
On
top
of
it.
B
B
What
else
view
the
source,
as
you
can
also
and
I?
Do
you
add
access
restrictions
to
your
namespaces
in
communities,
for
example
in
this?
In
this
example,
in
my
mixed
workload,
namespaces
the
project
pack
and
grid
engine
and
protract
and
production
are
allowed
to
submit
their
their
workload
into
this
namespace
again,
you
can
deny
something
for
you
can
deny,
am
accessing
that
mixed
workload.
Namespace.
You
are
also
able
to
to
limit
the
hosts
to
d2,
which
D
D
cube.
Anita's
workload
should
go
to
in
my
example.
A
C
So,
first
of
all,
thank
you
so
much
for
that
presentation
and
demo.
So,
if
you're
already
familiar
with
this,
the
way
that
we
do
the
QA
is
you
send
me
a
private
message
on
the
group
chat
and
I
will
read
it
out
loud
in
order.
So
if
you
click
on
the
chat
feature
at
the
bottom
of
your
screen,
you
can
select
my
name,
which
is
online
Meetup
and
you
can
send
me
any
questions.
You
have
and
I'll
read
them
out
loud.
C
A
So
that's
defined:
could
you
open
up
one
of
the
llamo
files
for
one
of
your
your
controllers,
so
I
brought
this
up
a
little
bit
when
we
were
when
I
was
doing
my
presentation,
and
it's
really
it's
through
there's
a
couple
of
different
ways.
So
the
way
that
we're
using
this
in
the
demo,
you
can
see
there
in
the
labels
section
about
the
tent
flying
down.
It
says
nav
outside
AO,
slash
project,
so
that's
a
special
label
that
nav
ops
command
is
actually
looking
for.
A
So
that
allows
this
this
specific
pod
to
be
bound
to
that
project.
We
also
support-
and
we
didn't
necessarily
show
it
in
the
UI,
but
I
think
Stephan
can
pull
it
up
right
here.
We
have
a
thing
called
application
profiles
down
in
the
configure
section.
Oh
we
don't
have
any
in
there
right
now,
so
yeah
perfect.
A
So
we
also
have
application
profiles
which
allows
you
to
add
a
whole
bunch
of
let's
say
labels
or
annotations
that
you
could
add
manually
or
individually
to
a
pod,
allows
you
to
group
them
under
this
one
kind
of
consistent
application
environment
description
which
can
this
can
then
be
attached
to
the
to
the
pod
instead.
So,
for
example,
you
see
on
here
down
at
the
bottom
there's
a
project
item,
we
could
create
a
new
application
profile.
A
So
Federation
is
a
very
interesting
concept
for
us
right
now.
In
general,
we
don't.
We
haven't,
really
been
approaching
the
Federation
style
of
growing
an
environment,
so
right
now
we're
we
would
generally
look
for
a
single
cluster
as
opposed
to
a
federated
cluster.
If
there's
some
I,
we
could
take
some
offline
discussion
with
the
people
who
are
really
interested
in
Federation,
but
it's
something
that
we
discussed
quite
a
bit
internally
but
I
guess
to
directly
answer
the
question
right
now.
This
is
not
targeting
federated
clusters.
A
Whatever
pod
happens
to
get
in
when
the
quota
actually
goes
from
being
in
violation
or
being
hit
which
everyone
tries.
Next
is
the
one
that
gets
in,
and
that's
likely
going
to
be
from
some
controller
that
you
have
running
in
the
system,
so
that
really
makes
it
difficult
to
be
able
to
you
know
consistently
or
predictably
manage
which
workloads
are
gonna
run
next,
when
you're
running
into
any
type
of
quota
or
limit.
So
really
where
command
will.
Let
you
you
operate
is
in
the
the
case
where
you
will
have
lots
of
quotas.
A
You'll
want
to
make
sure
that
the
important
work
runs
properly
in
those
environments.
Our
quotas,
like
I,
said
they're
runtime
quotas.
So
what
you
would
see-
and
maybe
you
picked
up
in
the
demo-
is
there
would
be
pods
that
would
be
pending
and
sitting
around
pending
waiting
to
run
and
that's
because
they
are
scheduled,
won't
schedule
them
because
they're
violating
the
runtime
quota.
So
so
those
are
two
things
right
there
and
we
believe
those
two
are
very
important
in
order
to
achieve
multi-tenancy
when
wanting
to
share
the
resources
efficiently
across
your
teams
and
efforts.
A
Right
now,
the
workloads
that
we
were
running
in
there
actually
will
run
to
completion
workload,
so
they
were
and
it's
once
the
capacity
was
available.
The
higher
priority
jobs
after
the
slider
change
took
over.
So
we
are
working
on
in
the
next
version
of
the
system
to
be
able
to
do
preemption
and
eviction
of
workloads
based
on
violations
of
the
policy,
but
that
wasn't
shown
in
this
in
this
demo.
C
A
That
is
in
the
product
that
we
have
currently
under
development,
for,
as
in
addition
to
the
nav
app
suite
called
nav
ops
control.
We
have
it
as
part
of
our
core
Grid
Engine
project,
so
the
way
that
I
would
describe
this
is
we
would
know
our
scheduler
keeps
track
of
all
of
the
requests
for
work
that
are
made,
and
it
will
record
this
into
a
reporting
and
accounting
file
through
that
file.
We
can
see
you
know,
what's
being
requested
and
what's
being
run
and
determined
why
things
were
in
being
able
to
run.
A
What
we
we
do
not
necessarily
detect
the
current
consumption
from
other
schedulers
at
this
point.
This
is
something
that
we
have
as
part
of
our
our
pretty
much
next
release
of
command,
where
we'll
be
adding
more
resource
collection
to
be
able
to
detect
the
consumption
of
of
the
resources
that
we
care
about.
So
the
the
the
answer
to
the
question
right
now
is
no.
The
command
is
not
aware
of
consumption
of
resources
by
other
schedulers,
but
that
is
on
a
near-term
roadmap.
A
Yes,
there's
there's
no
reason
why
this
wouldn't
work
with
Haq
Bernays
master's,
the
only
interface
that
command
has
with
kubernetes
is
via
the
api
server,
which
we
discover
using
the
standard.
Kubernetes
go
client
libraries.
So
as
long
as
that
is
failing
over
properly,
you
know
if
you
have
an
H,
a
failover
with
your
H,
a
master's.
A
It
should
be
fine,
because,
literally,
what
we're
using
is
the
standard,
kubernetes
client
to
do,
gets
and
watches
of
the
kubernetes
api
api
models,
as
well
as
then
running
kind
of
the
stock
kubernetes,
a
binding
command
to
bind
a
pod
to
a
node.
So
if
both
of
fundamentally
driven
through
the
core
kubernetes
go
client,
libraries.
C
A
A
So
so,
yes
definitely
so,
so
what
would
happen
is
the
way
that
would
actually
happen
is
when
you
like
Gordon,
a
node,
for
example.
It
would
trigger
a
kubernetes,
actually
manages
manages
that,
so
what
would
happen
is
you
would
chord
in
the
node
and
those
pods
would
get
removed,
as
you
know,
as
appropriate
based
on
kubernetes,
you
know,
changing
the
you
know,
deleting
the
actual
pots,
and
then
command
would
see
that
the
pods
were
deleted
and
release
those
resources,
and
then
you
know,
as
as
they
get
thrown
back
on,
as
the
controllers.
A
Try
to
recreate
the
deleted,
pods
command
would
see
those
and
place
them
based
on
the
current
state
of
the
cluster
and
the
current
current
policy
configured.
So
you
know
they
would
get
put
on
different
machines,
and
there
you
put
on
different
machines
based
on
you
know
their
priority.
They're
currently
calculated
priority,
as
well
as
the
resources
available
in
the
cluster.