►
From YouTube: Running Mixed Workloads on Kubernetes at the Institute for Health Metrics and Evaluation
Description
Running Mixed Workloads on Kubernetes at the Institute for Health Metrics and Evaluation - Dr. Tyrone Grandison, Institute for Health Metrics and Evaluation (IHME), University of Washington
The mission of the IHME is to apply rigorous measurement and analysis to help policy makers make better decisions on a range of health policy issues. Like other organizations, the IHME have embraced containers and micro-services aggressively to better support hundreds of collaborating researchers.
In addition to containerized workloads, the IHME run a wide-variety of traditional analytic, simulation and high-performance computing workloads on an HPC cluster with 15,000 cores and 13PB of storage. Researchers increasingly need to combine both containerized and non-containerized elements into workflow pipelines, and a key challenge has been ensuring SLAs for various departments and avoiding duplicate infrastructure and unnecessary data movement and duplication. In collaboration with industry partners, IHME have deployed a unique solution based on Univa’s Navops technology that allows them to combine containerized and traditional analytic and high-performance application workloads on a single shared Kubernetes cluster, ensuring departmental SLAs and helping contain infrastructure costs.
In this talk Dr. Grandison will discuss IHME, their experience deploying containerized applications and how they went about using Kubernetes to support a variety of new containerized applications as well as a variety of traditional analytic applications.
About Dr Tyrone Grandison
Tyrone is the Chief Information Officer leading the IT team at the IHME, independent global health research center at the University of Washington. The IHME provides rigorous and comparable measurement of the world’s most important health problems and evaluates the strategies used to address them. Prior to his involvement at the IHME Tyrone was Deputy Chief Data Officer at the U.S. Department of Commerce in Washington D.C. and a Presidential Innovation Fellow assisting with the CENSUS’ CitySDK projects and project at the US Department of Labor. Prior to this, Tyrone held various positions at IBM’s Almaden Research where he achieved recognition as a Master Inventor. Tyrone holds a PhD in Computer Science from the Imperial College of London, a MSc in Software Engineering from University of the West Indies and is a graduate of the IBM T J Warson Research Center micro MBA program in business.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
So,
first
of
all,
thank
you.
Everyone
for
coming
out.
I
know
this
is
almost
lunchtime
when
you
could
be
doing
better
things,
so
choice
for
you.
We're
gonna
talk
about
mixed
workload,
so
that's
basically
high-performance
computing
workloads
and
container
workloads
and
experiment
that
I've
been
doing.
We've
been
doing
a
Taichi
me
with
you,
neva
I'm,
Tyrone
Granderson.
If
that
wasn't
clear,
I'm
the
CIO
for
Aicha
me
just
started
a
few
well
year
and
a
half
ago
there.
This.
B
A
A
It's
awesome,
there's
two
more
than
I
ever
have
anywhere
else,
that's
perfect,
so
this
is
gonna,
be
a
quick
intro,
so
I
Jimmy
is
nonprofit.
So
it's
based
in
the
School
of
Medicine
at
University
of
Washington.
We
focus
on
population,
health,
sometimes
called
public
health
and
we
look
at
it
across
the
globe.
So
every
some
country
in
the
world
and
we
collect
data
from
censuses
surveys,
scientific
papers.
A
A
We
normally
produce
it
in
one
a
data
set
to
some
visualizations.
We
provide
training
and
we
do
special
analysis
for
the
World
Bank,
the
World
Health
Organization.
We
do
county-by-county
analysis
for
different,
like
I
said
countries,
for
example,
India
and
China.
We
also
do
some
work
for
the
sustainability
sustainability,
Development
Goals,
that
always
catches
me,
and
that
is
with
with
help
from
the
Gates
Foundation,
who
is
a
big
funder
of
ours.
Our
customers
tend
to
be
researchers,
so
internal
thme
and
external.
We
have
a
huge
community
of
epidemiological
researchers
that
depend
upon
us.
A
We
want-
and
we
actually
do
currently
work
with
some
advocates,
so
people
that
have
special
interests
in
particular
topics
you
know,
say:
you're
interested
in
the
impact
of
smoking
on
the
population
of
Peru,
or
you
know
you
may
be
concerned
about
the
effect
of
diarrhea
on
the
livelihood
of
of
kids.
I
mean
all
fun
topics:
policymakers,
the
people
that
actually
make
laws
that
want
to
see
the
communities
that
they
serve
actually
improve.
A
We
work
with
them
too,
and
if
you've
seen
a
story
in
the
news
on
financing,
Global
Health
on
sustainability,
Development
Goals
on
a
topic
that
actually
touches
on
a
data
aspect
of
Public
Health,
the
source
for
that
data
is
normally
our
system.
Our
hme,
the
academics
and
the
researchers
tend
to
be
our
largest
segment
right
now,
and
the
reason
why
I'm
here
and
I'm,
going
around
to
other
developer
conferences,
is
that
to
get
the
word
out
that
you
know
we're
looking
for
people
to
use
our
data
for
multiple
different
things
at
the
local
level
right.
A
So
all
this
data
is
free
for
non-commercial
use.
At
this
point,
so
I
Jimmy's
process
is
pretty
simple.
I
mentioned
before
we
have
multiple
different
data
sources.
Each
year
we
basically
run
Ron,
meaning
I
could
look
at
150
to
around
300,000
data
sources.
We
basically
transform
those
transform
them
multiple
ways,
standardizing
them.
According
to
icd-9
icd-10,
which
are
sorry
disease
codes
for
healthcare,
we
then
actually
do
some
checks
and
remove
some
garbage
codes.
For
example,
you
don't
expect
you
know
men
to
have
a
higher
risk
of
something
based
upon
pregnancy.
A
So
you
know
those
associations
have
to
go
once
it's
actually
like
transformed.
They
then
become
inputs
to
our
analysis
process
and
that's
where
we
have
around
250
at
this
point,
researchers
internally,
that
runs
artistical
models
on
the
data
itself,
so
they
one
use
existing
off-the-shelf
models
to
they
modified
m3/day
figure
out
like
what
gaps
we
have
in
the
data
currently,
because
we
have
to
do
it
across
all
countries
in
the
world
and
we
one
models
based
in
a
continual
basis.
A
We
can
probably
run
for
the
entire
global
burden
of
disease
process,
which
is
the
data
set.
We
produce
run
models
at
least
six
to
seven
times,
so
we
have
our
environment
in
the
back
end,
always
going
once.
We
actually
have
what
people
call
an
intermediate
result.
We
then
go
what
we
can
get
a
collaborator
network
and
the
collaborator
network
is
around
3,000
researchers
across
the
world
that
then
vet
that
the
numbers
actually
look
good
and
they
help
us
modify
the
models
themselves
so
highly
collaborative
process.
A
At
one
point,
we
come
to
a
point
where
we
say
this
is
good
enough,
and
then
we
call
that
outputs
right
and
the
outputs
we
publish
in
everything
from
the
science
journal
to
well,
essentially,
all
the
person's
journals
and
the
hub.
The
color
spaces,
where
we
publish
to
we
actually
put
these
policy
papers
for
different
countries
and
our
global
engagement
team
within
I
to
me
actually
actually
works
on
stories
with
the
media
to
kind
of
highlight
the
the
issues
that
are
actually
in
the
data
itself
that
we
produce
so
in
terms
of
exact
artifacts.
A
I
just
mentioned
the
Goldberg
disease,
and
what
this
is
is
a
study
of
everything
of
injuries,
of
risk
factors
of
diseases
that
actually
cause
people
to
lose
years
on
their
life
right.
We
have
metric
that
we
call
the
disability
adjusted
life
years,
that
we
use
to
compare
multiple
different
phenomena
across
the
entire
spectrum.
A
So
it
could
be
car
crashes
in
Lima
versus
the
effect
of
smoking
in
Greece
versus
number
of
murders
in
you
know
bigger
country
Zambia,
and
we
do
that
across
age,
sex,
geography
and
over
time
over
time,
meaning
from
1990,
because
we're
not
that
good
right-
and
this
leads
to
something
that
we
call
the
the
dhdx,
which
is
a
global
health
data
exchange.
Where
we
put
every
single
source
that
we've
actually
used
in
one
place.
A
Added
to
that,
what
we're
focusing
on
now
is
what
we
call
future
health
scenarios.
So
how
do
we
in
so?
How
do
we
tell
a
country
or
an
individual
or
community
member,
that
for
your
community
in
60
years?
This
is
what
what's
gonna
actually
be
the
biggest
factors
for
you.
These
are
the
things
that
you
should
be
worrying
about.
That's
that's
the
forecasting
work
for
that.
Of
course,
we
have
to
do
multiple,
different
scenarios.
You
know
one
optimistic
one
pessimistic
one.
A
You
know
given
the
trajectory
on
this
is
what's
going
to
happen
and
I
mentioned
before
we
do
spatial
analysis
both
on
geographies.
We
did
one
on
India
just
two
weeks
ago,
the
India
launched
and
we
do
specific
ones
for
China
and
we
have
a
roster
of
11
per
year
data
coming
out.
So
that's
in
terms
of
what
we
produce
like
that.
Is
it
quick
example.
A
We
just
launched
in
February
February,
8th
the
global
burden
of
disease,
2016
version,
and
we
do
this
every
single
year,
so
we
have
a
recover
335
different
diseases.
We
cover
like
a
thousand
different
sequelae
of
disease,
who
here
knows
what
sequelae
means
awesome
good.
So
sequelae
is
a
consequence
of
a
disease,
so
something
that
happens
as
a
result
of
right.
I
won't
go
into
this,
so
risk
factors
cause
are
things
that
cause
diseases
cause
risk
peers.
So
we
have
quite.
We
have
multiple
different
facets
to
the
day
we
produce.
We
have
it.
A
Over
a
number
of
years
we
had
around
three
billion
data
points
that
we
ingested.
The
outputs
data
itself
was
only
30
terabytes.
The
input
date
and
intermediate
results.
You'll
see
it
it's
in
the
petabytes.
We
have
3000
points
of
metadata
and
you
know
last
year
this
is
rough.
We
probably
use
their
own,
like
175
thousand
data
sources
right
to
get
there
in
terms
of
what
we
produced
for
it.
It
was
very
simple:
we
produced
the
reports
in
the
land
set,
that's
the
main
publication.
A
A
A
We
basically
work
with
a
China
collaborative
Center,
really
well
a
good
example
of
like
how
how
we
normally
do
impact
on
the
policy
space
would
be
us
working
with
the
Iran
and
Ministry
of
Health
right
and
having
them
look
at
the
data
and
analyze
it
and
seeing
that
in-house
air
pollution
was
a
number
one
factor
in
in
having
their
people
not
live
as
long
as
they
should
right.
So
the
average
life
expectancy
back
in
1990
was
around
44
or
45.
A
The
government
started
a
program
they
developed.
You
know
clean
stoves
that
burn
more
efficiently,
didn't
release
so
many
toxins
within
the
space
of
16
years.
They
basically
got
their
life
sh-t
up
to
like
63
right.
So
that's
this
them
seeing
the
data
figuring
out
like
what
it
takes,
that
you
solve
that
problem
and
just
implementing
it
and
seeing
how
it
works
over
time.
So
with
that
I'm
going
to
pass
it
over
to
Jason
talk
about
you,
neva.
B
Yeah
thanks,
okay
cool,
hear
me
yeah
awesome,
so
who
is
Unova
right,
leading
innovator?
We
actually
are
working
with
many
different
partner
organizations
to
help
customers
really
figure
out
the
best
approach
for
running
their
applications
on
top
of
kubernetes.
We
have
many
different
features.
They
come
along
with
our
product
and
we're
based
in
Chicago
with
offices,
also
in
Canada
and
Germany.
B
We
have
many
different
customers
right.
You
guys
probably
don't
care
too
much,
but
a
few
different
verticals
there
as
far
as
customers
and
what
they
do.
A
lot
of
these
enterprise
customers
run
many
different
analytical
batch
jobs,
which
is
what
Tyrone
will
talk
about
in
a
little
bit
here
and
then
finally
kind
of
our
product
that
we're
talking
about
nav,
ops,
so
nav
Ops
has
many
different
features
that
it
enables
within
kubernetes.
Some
of
these
are
virtual
multi-tenancy.
B
We
actually
offer
mixed
workloads
right
and
be
able
to
run
your
batch
jobs
on
top
of
kubernetes,
which
is
kind
of
the
whole
point
why
you
guys
are
in
this
room,
so
we're
gonna
get
into
that
in
just
a
little
bit
here
and
then
also
other
things
right.
So
you
can
set
priorities
for
your
applications
and
then,
depending
on
the
resources
that
are
available
and
what
the
priorities
are
on
those
applications.
Kubernetes
will
now
be
able
to
run
your
higher
priority
apps
before
your
lower
priority
apps,
which
is
not
currently
built
into
the
default
scheduler.
A
Can
you
hear
me
perfect
not
that
interesting,
but
we'll
see?
So,
let's
start
talking
about
item
e
and
a
reason
why
we
decided
to
go
along
go
on
this
path.
This
journey
with
with
univille
I'm
gonna,
start
off
with
the
boring
stuff
and
go
through
this
quickly.
The
team
itself
is
here
to
actually
make
sure
that
both
the
collaborators
and
the
researchers
within
I
to
me
cannot
to
do
their
jobs
and
cannot
you
do
it
with
a
mindset
to
innovation
right
right
now
the
team
is
61
people
and
we
basically
cover
everything.
A
So
we
run
our
own
cluster
environment.
I'll.
Tell
you
more
about
that
later
on,
we
control
all
the
data
we
have
over
a
hundred
different
database
servers
in-house.
We
do
or
one
Volition's.
We
have
a
data,
science,
engineering
and
kind
of
desktop
and
talk
to
him
when
it
comes
to
our
users.
So
we
have
two
main
users
right
and
the
first
set
is
the
researchers
and
researchers
have
one
well
several
things
so
background,
so
they
have
different
technical
backgrounds
right.
A
B
A
A
They
tend
to
want
to
actually
write
a
lot
of
statistical
medalists
from
scratch,
which
is
a
blessing
and
a
curse
all
at
the
same
time,
and
they
want
to
customize
like
what
didn't
want
to
run,
because
some
of
them
are
actually
on
the
cutting
edge
and
will
use
whatever
the
latest
thing
is
out
there,
even
though
it
may
not
be
supported
by
my
team
all
right.
So
the
the
task
here
is
that
you
have
a
bunch
of
really
smart
people
that
are
one
academics
right.
A
That
may
have
adequate
training
as
programmers
computer
scientists
that
are
running
jobs
and
the
auto
off.
You
know:
I
have
one
job
that
spawns
off
ten
two
hundred
thousand
other
jobs
that
runs
on
probably
100
to
200
different
data
points
and
actually
does
a
lot
of
like
IO
right.
That's
that's!
The
high
performance
computing
need
from
the
research
org.
A
We
also
have
a
particular
kind
of
researcher
that
you
know
after
they've
done
their
processing
and
have
their
models
to
a
point
where
they're
happy.
They
want
to
have
a
web
server
and
a
way
touch
interact
with
the
outside
world,
and
you
want
to
save
an
environment
where
they
can
actually
do
that
right.
So
two
needs
there.
One
is
obviously
we
have
to
support
the
high
performance
computing
and
obviously
we
need
to
support
them.
Having
services
that
stand
up
by
themselves
and
necessarily
don't
don't
fail,
but
they're
not
really
too
upset
if
they
do
fail.
A
The
second
set
of
group
uses
that
we
have
is
is
the
support
functions
right,
so
the
first
set
is
what
I
call
the
business
critical
mission,
critical
user
group-
and
the
second
is,
you
know
the
finance
to
HR.
The
global
engagement
means
marketing
here
and
when
we
produce
for
them.
Is
this
storage
right,
so
document
management
for
them
collaboration,
management
and
customer
relationship
management,
because
we
do
have
to
interact
like
I
said
before,
with
3,000
collaborators
in
some
sort
of
a
seamless
fashion,
and
that
is
not
easy
in
the
current
environment.
A
A
Hpc
notes,
mix
of
Intel
and
AMD
I
have
a
description
of
one
of
the
newer
Intel
notes
that
we
just
got.
We
run
vSphere.
We
have
at
this
point,
probably
over
400
VMs.
We
have
darker
and
rancher
in-house
300
containers
in
terms
of
like
active,
daily,
available
storage
that
can
be
used.
We
have
5.8
petabytes
and
tape
storage,
because
tape
is
never
gonna
die
ever
ever
because
it's
so
cheap.
We
back
everything
up
and
we
basically
have
in-house
9.2,
petabytes
and
access
to
God
knows
how
many
in
North.
A
Alright,
so
a
little
bit
more,
so
the
HPC
cluster
itself
has
two
main
purposes.
We
do
primary
modeling,
we
do
machine
learning,
we
have
storage
tiers
that
use
store,
necks
NetApp,
we
are
a
big
cumulant.
Kimmel
allows
us
and
we
use
InfiniBand
and
fibre
channel
tachi
connect
everything
in
terms
of
software.
You
know
before
starting
at
I.
To
me
it
was
the
wild
wild
west
and
everyone
was
using
everything
after
still
going
on,
but
the
path
is
that
we
want
to
actually
take
everyone
down
the
our
Python
path.
A
So,
right
now
we're
saying
for
primary
modeling,
you
want
to
support
our
so
our
studio,
our
shiny.
We
want
people
to
not
have
to
mess
with
submitting
jobs,
like
my
name
at
the
command
line,
which
is
the
current
de
facto
right.
So
we
use
jupiter
notebooks
for
that,
of
course,
Python
numpy,
pandas,
yada
yada.
C
A
What's
going
on
with
the
jobs
in
terms
of
CI
CD,
we
use
Jenkins,
primarily
Luigi
to
small
extent
and
we're
also
building
our
own
customed
workflow
management
software,
which
we're
going
to
release
back
to
the
open
source
community
in
the
next
year,
simply
because
our
environment
is
kind
of
different
and
harsh
from
the
existing
tools
that
are
out
there
right
now.
I
can
talk
more
about
that
offline.
That's
fine
for
databases
is
Percona,
my
sequel
or
a
DB
column
store
and
we're
looking
at
what
the
future
is.
A
So
how
do
we
actually
move
off
these
environments?
Because
these
environments
don't
actually
support
all
the
scenarios
that
we
need
terms
of
the
queries
that
we
have
to
ask
and
in
terms
of
where
we
are,
you
know,
standard
HTML
CSS,
we
are
big,
react,
GS
house
and
we
we
like
to
develop
stuff.
We
like
to
grow
stuff
inside,
so
we
have
something
called
I
Jimmy
UI
on
github,
again
it's
open
source
and
it's
free
go
check
it
out
most
of
the
components
we
actually
build.
A
That
are
online,
that
this
of
that
org
is
built
using
hme,
UI,
alright,
so
current
architecture
in
one
horrible
diagram
is
25,000
cores
most
of
the
users
that
you
use
a
command
line
to
access
or
shared
storage,
and
we
have
a
really
really
fast
connection
between
or
storage
and
we
mode
everything
on
the
particular
nodes
themselves.
Alright,
so
that
was
a
boring
part.
The
part
that
that
is
interesting
is
this.
Like
we
have.
This
need
Tachi,
moved
to
a
new
environment.
A
What
we
have
an
investment
I
have
investment
in
uge
because
we've
been
using
it
for
10
years
now.
Well,
actually,
we've
been
using
for,
however
long
it
got
rebranded
to
ug.
What
ever
was
before
SG
some
form
of
Oban
using
so
the
researchers
are
familiar
with
it
right
when
we
got
in
there
and
we
actually
started
looking
at
we're
using
the
environment
realized
that
we
were
under
utilizing
a
lot
of
resources
and
at
peak
times
people
were
complaining
and
not
getting
resources.
A
A
We
got
opportunity
at
she
start
fresh,
so
we
got
opportunity
to
actually
make
a
huge
purchase
in
HPC
notes
and
use
this
as
a
forcing
factor
to
start
talking
about.
How
do
we
move
to
an
environment
that
supports
both
containerized
workloads
and
HPC?
So
we
bought
the
nodes.
We
started
a
plan
where
we
said:
ok,
we're
gonna
actually
put
kubernetes
in
from
scratch,
which
was
a
luxury
luxury
for
most
companies
and,
let's
figure
out.
B
All
right
so
I
mentioned
a
few
of
the
different
features
that
nav
Ops
has.
One
of
those
is
the
virtual
multi-tenancy
which
actually
allows
you
to
share
your
clusters
across
teams
and
applications,
and
then
you
have
more
control
on
what
they
have
access
to
and
what
the
resources
are
available
for
those
as
well
as
mixed
workloads.
B
This
is
kind
of
how
we
integrate
with
kubernetes
right,
so
we've
got
the
regular
kubernetes
architecture
we
come
in
and
we
just
replace
the
default
scheduler
and
then
you
can
actually
set
up
in
your
llamo
files
using
annotations
or
whatever,
to
tell
it
to
actually
use
our
scheduler
rather
than
the
default
kubernetes
scheduler,
but
that
default
community
scheduler
still
runs
in
the
background
right.
So,
if
anything
does
happen,
those
jobs
get
passed
to
that
default,
scheduler
as
well.
So
there's
no
like
single
point
of
failure
or
anything
there.
B
B
You
can
set
the
priorities,
lots
of
other
things
that
you
can
do
to
make
sure
that
you're
configuring
our
tool
correctly
and
then
it's
just
a
simple
pod
installation,
so
cube
CTL,
apply
point
to
your
Yama
file
done
what
it
does
is
it
allows
you
to
have
advanced
policies
for
kubernetes,
so
here
we
can
set
up
our
work,
lured
priorities.
We
can
have
different
proportions
and
we
can
actually
adjust
those
on
the
fly
and
if
you
are
a
UI
person,
you
can
go
in
and
drag
this
around
right
and
it'll
automatically.
B
Take
resources
within,
for
example,
batch
workloads
and
move
those
over
to
development
or
you
can.
If
you
don't
want
to
use
a
UI,
you
can
do
that
easily
through
the
CLI
as
well,
and
you
can
also
automate
that
within
your
workflow
or
anything
else,
and
then
once
you
enable
and
you
actually
activate
command.
We're
gonna
have
some
type
of
a
solution
like
this,
which
is
what
we
provided
for
IHM
II,
and
here
you
can
see
that
we've
got
command
running
on
top
of
kubernetes
and
then
we've
got
our
Grid
Engine
stuff
running
the
batch
workloads.
B
On
top
of
that
as
well
as
you
can
see,
we've
got
container
a
and
container
B
right,
different
colors
there,
and
we
can
run
those
all
on
top
of
the
same
shared
cluster,
which
then
leads
to
much
better
resource
usage
in
utilization.
So,
prior
to
setting
this
all
up,
they
were
actually
below
20%
utilization,
lots
of
resources
that
actually
being
used
and
they
were
just
being
wasted
and
then,
after
gain,
everything
set
up,
we're
actually
over
50%
utilization,
which
is
fairly
good
for
an
always-on
cluster.
A
Okay,
so
this
is
very
simple,
so
we've
been
really
happy
to
work
with
you,
neva
and
rolling
this
out,
and
just
comment
that
this
is
still
you
know,
pilot
in
my
eyes
that
we're
still
working
through
and
we
love
the
fact
that
it's
simplified
our
administration
and
it's
that
she
made
our
environment
actually
really
more
flexible
and
with
that
I'm
just
gonna
end
it
there
and
give
people
time
for
questions.
If
we
have
time
for
questions.
B
Well,
so
nav
ops
runs
on
any
kubernetes
deployment
right,
it's
just
a
pod
that
you
deploy
so
as
far
as
VMware
it
goes.
We
don't
do
anything
on
the
VMware
side
right
if
you
can
run
kubernetes
on
top
of
it,
whether
it's
Rancher,
like
I
hme,
is
using
or
core
OS
or
some
type
of
OpenShift
something
you've
rolled
yourself.
Even
you
just
deploy
our
application
on
top
of
it
and
it
enables
those
additional
features.
Oh.
A
A
B
Yeah,
so
as
far
as
command
goes,
it's
actually
not
open-source.
You
can
download
it
and
run
it
on
to
five
nodes
right
for
free
and
give
it
a
shot
and
see
what
you
think
about
it.
If
you
are
doing
it,
we
have
some
other
open-source
projects
out
there
right.
So,
if
you
are
interested,
you
go
to
github.com,
slash,
Unova
and
see
what
we've
got.
B
One
of
them
is
like
a
broker
for
running
mace
those
frameworks
on
top
of
kubernetes,
so
if
you're,
using
Hadoop
or
Kafka
or
anything
like
that
applications
written
in
that
you
can
use
you
RB,
which
is
our
broker
and
just
run
those
straight
on
kubernetes
right.
You
have
to
have
multiple
clusters,
but
as
far
as
command
goes,
it's
not
currently
open
source,
but
we're
looking
into
something
along
those
lines.
B
C
Work
at
Unova
been
there
quite
a
quite
a
while
done
a
lot
of
work
with
with
the
grid
engine
part
and
so
yeah
we
have.
We
do
have
preemption
in
in
net
ops
command
and
it's
following
some
of
the
policies
and
work
we've
been
doing
in
our
HPC
scheduler
in
general.
You
know
what
we
always
say:
preemption
is
a
great
thing,
but
we,
you
know
it's.
It's
always
really
good:
to
try
to
get
to
environments
where
you
don't
have
to
do
it
and
that's
what
we
really
try
with
command.
C
You
know
we
understand
you
look
how
kubernetes
clusters
are
growing,
there's
tons
of
container
events
a
day
which
gives
the
scheduler
chances
to
fit
the
cluster
utilization
against
the
policy
without
having
to
preempt
work.
That
being
said,
we
do
have
well.
I
really
need
my
thing
to
run
and
we'll
kick
something
out
and
let
this
really
important
thing
get
going
on.
A
A
Right
now,
we
are
experimenting
with
Kafka
as
primary
source,
which
we're
trying
to
at
URI
architect
everything
at
this
more
in
touch.
We
have
the
data
not
move
as
much
and
have
the
compute
move.
So
as
interim
the
simplest
thing
is
what
we
do,
but
we
I
don't
think
that's
gonna
survive
long
long
term.
A
C
Yep,
so
it's
definitely
I
know
it's
an
alpha
feature
that
was
made
part
of
of
1.8
I.
Think
one
of
the
big
things
that
command
is
really
doing
from
a
prioritization
perspective.
That's
still
not
not
in
kubernetes
is
you
know,
Cabernets
has
the
ability
for
you
to
set
a
priority
on
a
pod
and
say
this
is
a
high
priority
pod.
But
that
say
you
know
it's
somebody
setting
that
you
know.
So
you
create
your
classes.
C
You
could
have
some
some
policy
that
sets
what
commands
really
about
is
dynamically
calculating
over
time
the
priorities
of
your
pods,
which
can
change
based
on
config
reconfiguration
of
the
scheduler
additions
of
new
workloads.
So
it's
it's
redone
every
15
seconds,
all
the
pods
that
we
prioritize
based
on
your
current
policy.
So
yes,
there's
definitely
convergence
I!
Think
in
the
computerized
community.
We've
actually
been
thinking
about,
maybe
doing
something.
That's
much
closer
to
the
kubernetes
schedule
that
that's
pretty
much
just
been
manipulating
the
priority
field.
You
know
like
with
a
controller
or
something
so
there's.