►
From YouTube: OpenShift Commons Briefing #140: The Path to Cloud Native Trading Platforms Jeremy Eder Red Hat
Description
The Red Hat Performance Team, along with our partners Solarflare and Supermicro, have been working together to leverage the latest technologies and features in the container orchestration space to demonstrate that it is possible to containerize extreme low-latency applications without any degradation in performance. The team used the well-known STAC-N1TM benchmark (https://stacresearch.com/nio) from STAC® (the Securities Technology Analysis Center), to prove out the technology.
STAC-N1 is a financial services-focused benchmark which focuses on how quickly applications in the trade flow (algorithmic “black boxes”, matching engines, smart order routers, etc.) can get information from and to the network. The STAC-N1 benchmark suite measures the performance of network stacks under a simulated market data environment using a convenient, software-only test harness.
It’s our belief that Linux containers and container orchestration engines, most notably Kubernetes, are positioned to power the future of enterprise applications across industries. Along the way, challenges to this perception are bubbling up. In the area of performance-sensitive workloads, like portfolio risk analysis and
other financial transactions where a matter of microseconds can mean the difference between success and failure, there are strong concerns. This is why Red Hat has embarked on a mission to enable Red Hat OpenShift as a performance-sensitive application platform (P-SAP) to better
support these critical workloads.
While the STAC-N1 benchmark is important, we believe this effort can have an impact beyond financial services. Artificial intelligence, machine learning, high
performance computing, and big data, to name a few areas, are all examples of workloads where the set of Performance-Sensitive Application Platform features are critical.
A
Hello,
everybody
and
welcome
again
to
another
OpenShift
Commons
briefing.
This
time
we
have
Jeremy
eater
from
Red
Hat
who's
going
to
talk
to
us
about
the
past
child
needed
trading
platforms,
and
this
comes
out
of
some
of
the
work
he's
been
doing
with
a
red
hat
performance
engineering
group
and
a
lot
of
interest
in
this
topic.
A
B
My
name
is
Jeremy
eater
yep
I
work
in
the
Red
Hat
performance,
engineering
team
and
they're,
not
OpenShift.
What
I
have
today
is
a
presentation
on
how
we
can
use
some
of
the
new
features
in
kubernetes
and
in
OpenShift
to
deploy
a
benchmark
that
is
focused
on
Wall,
Street
workload,
low
latency,
networking
and
essentially
the
the.
The
summary
is:
how
can
we
bring
some
of
those
high
performance
workloads
onto
onto
a
container
container
orchestration
platform?
B
So
I'll
start
with
that.
As
the
summary
and
one
interesting
thing
in
this
slide
comes
from
one
of
the
product
manager
supplies
from
from
Red
Hat
Summit
this
year.
It's
interesting
to
note
that
red
adoption
of
openshift
itself
is
is
pretty
hot
and
heavy
in
the
financial
world,
not
necessarily
for
these
super
high
performance
workloads.
Yet
because
we've
are
still
building
the
features,
but
that
ecosystem
is
interested
in
in
you
know:
productized
enterprise
ready
a
version
of
kubernetes,
which
is
what
open
chef
is
so
here's
some
logos
from
that
segment.
B
You
can
see
Financial
Services
represents
currently
over
one
third
of
the
sales
of
open
shift.
So
hopefully
this
talk
is
applicable
to
that
group
and,
in
addition
to
use
the
percentages
of
Business
at
Red
Hat's
event
in
San
Francisco
over
hats
in
in
May
there's
an
open
chef
Commons
day
that
preceded
the
summit
itself,
and
there
was
a
panel
related
specifically
to
financial
services.
B
Customers
during
that
open
shift,
Commons
gathering
and
one
last
thing,
I'll
mention-
is
that
Red
Hat's
participating
in
a
new
community
called
the
finis
community,
which
is
attempting
to
spearhead
more
usage
of
open-source
and
a
financial
services.
Vertical
dough,
quick
description
of
open
shift
and
the
unified
platform
with
core
LS.
Our
recent
acquisition
of
core
OS
at
the
end
of
January
2018
Red
Hat,
provides
a
full
end-to-end
stack,
including
the
operating
system,
kubernetes
distribution
operations,
framework
monitoring,
metrics
logging
and
so
forth.
Assemble
complete
a
complete
platform
to
run
your
applications
left-to-right.
B
You
can
also
modulate
open
ship
between
the
containers
of
service
platform
and
a
platform
as
a
service,
and
what
I'm
going
to
show
you
today
is
more
about
container
as
a
service,
rather
than
using
some
of
the
higher-level
D
ICD
tooling.
That
will
come
with
time,
but
is
just
not
part
of
this
demo,
so
those
some
of
those
more
platform-as-a-service
traditional
kind
of
features.
B
Okay.
So
what
the
backstory
here
is
that
we,
a
couple
years
ago,
identified
a
gap
in
not
Stream
kubernetes
or
in
the
container
orchestration
ecosystem
as
a
whole
in
addressing
running
some
performance,
sensitive
workloads
on
openshift
or
on
kubernetes,
and
this
pretty
little
diagram
on
the
right
is
the
extent
of
my
graphics
design
skills.
B
But
it
represents
essentially
that
some
survey
results
that
we
talked
to
a
quite
a
few
customers
and
different
industry
verticals
and
found
that
a
lot
of
them
who
are
running
applications
and
in
production
and
wanted
to
move
to
a
more
flexible
substrate,
had
a
lot
of
overlapping
requirements.
And
so
we
did
a
bunch
of
surveys
and
site
visits
and
a
lot
of
conversations
and
those
continue
to
this
day.
B
The
upstream
efforts
centered
around
creation
of
a
community
that
would
address
performance,
sensitive
applications
and
then
eventually
plumbing
all
of
those
features
in
a
way
that
helped
all
of
those
workloads.
At
the
same
time,
we
wanted
to
retain
what
makes
Kerber
Nettie's
great
and
that's
the
that
it's
it's
extremely
flexible
and
we
didn't
want
to
reinvent
some
of
the
existing
patterns
that
work
that
our
customers
mentioned
to
us
are
too
rigid
and
difficult
to
maintain.
We
wanted
to.
B
In
that
release,
we
have
three
new
features
that
relate
to
performance,
sensitive
applications
and
there's
a
blog
up
on
red
comm
that
talks
about
the
middle
in
a
little
bit
of
detail,
but
today,
I'll
go
into
a
little
bit
more
device,
plug-in
and
CPU
manager
and
huge
pages
support,
which
was
added
by
Derek
Carr
of
Red
Hat,
and
then
some
of
the
other
stuff
was
done
by
the
community.
Google
Intel
Nvidia
IBM
a
lot
of
community
participation
from
other
vendors
who
are
interested
in
running
similar
workloads
on
kubernetes.
B
A
couple
of
minutes
so
once
we've
got
that's
the
compute
and
memory
side
of
things,
of
course,
the
workload
also
has
you
know,
network
aspects
to
it
and
also
storage
aspects
to
it.
So
the
next
area
we're
focusing
on
is,
is
the
networking
side
and
one
of
the
key
things
understand
about
kubernetes
in
its
initial,
you
know.
The
way
it
is
upstream
now
is
that
each
pod
has
a
single
interface
and
all
pods
on
the
system
share
a
single
interface
in
terms
of
physical
interfaces
right.
B
So
what
we
wanted
to
do
was
help
our
customers
address
an
additional
set
of
use,
cases
that
were
like
vast
data
plane,
workloads,
whether
it's
telecom
or
distributed
deep
learning,
anything
that
or
video
streaming.
A
lot
of
those
workloads
have
huge
network
performance
requirements
and,
in
the
past,
have
used
a
variety
of
techniques.
B
At
least
this
presentation
has
links
to
the
ways
to
get
in
contact
with
or
to
participate
in
those
in
those
groups
which
I
would
definitely
encourage
anyone
listening
to
do.
In
the
last
year.
We've
done
a
couple
of
things.
I'll
show
demos
of
all
of
these.
In
a
few
minutes,
all
these
RPA,
except
for
sis
control
support
which
remains
in
tech
preview,
extended
resource,
is
not
intended
to
be
used
directly,
but
as
a
building
block
for
device,
plugins
and
I
will
show
a
demo
of
that
in
a
minute
as
well.
B
We
know
that
we're
not
done
there's
more
to
be
done.
We
have
overhead
in
the
ecosystem,
have
a
lot
of
work
on
our
plate.
One
of
them
is
handling.
You
know
hardware
topology
in
a
system
to
optimize
performance
even
further.
One
of
them
is
a
resource
API
to
to
allow
you
to
select
more
granularly
on
what
hardware
that
you
schedule
to.
B
For
example,
if
I
want
to
schedule
to
a
certain
model,
GPU
versus
another
or
certain
model,
Nick
versus
another,
that's
currently
not
possible
and
we're
gonna
try
and
make
that
level
of
granularity
possible
in
kubernetes
and
eventually
open
shipped.
We
have
a
multi
I
would
call
this
yeah
I
know
these
things
happen
overnight.
They
take
careful
design.
Another
one,
that's
ongoing
is
the
some
changes
around
the
existing
CPU
manager
to
make
it
friendly
to
ISIL
CPUs,
which
it
currently
is
not
and
I'll
share.
B
Some
benchmark
results
that
indicate
what
the
problem
is
there,
so
that's
the
platform
level
stuff
to
demonstrate
some
of
these.
Some
of
these
new
features,
Bunco
kosach,
who
I'm
not
sure
if
he's
joined,
I,
can't
see
to
blue
jeans,
but
he
led
this
project
to
onboard
a
financial
services
benchmark
into
OpenShift
Hong
Kong.
He
joined.
B
C
Yeah
sure
so
we
had
two
configurations.
We
started
with
a
bare-metal
to
have
some
some
baseline
and
we
got
this
new
solo
for
extreme
scale
adapters
with
Super
Micro
servers.
We
were
running
on
pair
metal
with
Red
Hat,
Enterprise
Linux
7
point
5.
After
we
established
a
baseline
of
bare
metal,
we
duplicated
a
complete
benchmark
on
openshift
the
same
hardware,
but
with
opposite
3.10.
B
B
Interestingly,
the
upstream
docker
I
did
not
accept
those
PRS,
so
I
don't
believe
they
support
them
now,
but
anyway,
those
pre
start
hooks
provide
the
kernel
bypass
libraries
into
the
into
the
container
painters
that
have
the
benchmark
in
them.
So
this
solar
flare
provides
as
a
network
adapter
vendor
that
provides
a
kernel
bypass
library
called
open
onload
and
in
order
to
mount
those
libraries
from
the
host
into
the
pod,
we
use
this
base
start
hook.
That
is
more
than
an
implementation
detail.
B
It
is
an
implementation
detail,
but
it's
an
important
part
to
know
about
what
it
takes
on
board.
These
the
use
of
these
kernel
bypass
adapters
into
into
OpenShift.
So
what
we're
doing
in
on
here
are
the
snowflake
nodes
towards
the
bottom.
This
is
the
logical
diagram
of
an
open
chef
cluster
topology,
as
you
can
colle
playing
in
the
infrastructure
nodes
and
then
you've
got
the
compute
nodes
and
potentially
a
storage
tier.
If
you
are
running
one,
the
snowflakes
on
the
bottom,
there
you
know,
that's
just
meant
to
represent
there's
something
special
about
those
nodes.
B
The
second
deployment
in
openshift.
This
is
a
logical
diagram
of
how
a
pod
gets
scheduled
are.
The
interesting
part
is
on
the
left,
blue
column
or
a
left
blue
box
there,
this
this
area
of
the
pods
back
section
of
an
existing
pods
back
that
we
created
the
resources,
feel
the
limits
field
now
contains
the
solar
flare
comm,
slash
at
the
SFC
:
1,
and
what
that
does.
B
Is
it
the
device
plug-in
surfaces,
the
number
of
Nick's
that
are
available
on
a
system,
so
it
discovers
the
hardware
and
then
it
publishes
that
to
the
capacity
and
allocatable
fields
within
the
node
stack
itself,
and
then
that
event
that
information
eventually
gets
up
to
this
to
the
cube
scheduler
itself,
so
that
when
a
pod
create
comes
in
that
requests,
one
of
these
devices,
the
cube
scheduler
knows
which
node
has
them,
which
node
has
some
available
and
and
then
schedules
them.
You
know
so
the
pod
ends
up
on
it.
B
Has
that
hardware,
then
the
cubelets
job,
the
atomic
OpenShift
node
processes
job
is
to
glue
those
things
together
is
to
take
the
hardware.
The
cgroups
do
the
secret
juggling
and
eventually
the
the
container
runtime
and
the
pre-start
hook
makes
sure
that
the
bypass
libraries
are
available
inside
the
pod
so
that
the
application
can
and
take
advantage
of
those.
Hopefully
that
all
made
sense
and
just
checking
a
little.
We
rejoined.
B
So
here
is
a
know:
here's
a
logical
diagram
of
the
benchmark,
pods
themselves.
We
had
two
physical
machines,
in
this
case,
a
producer
which
sends
traffic
and
a
consumer
which
reads
traffic
and,
in
this
scenario,
actually
reflects
it
back
to
the
producer.
So
it's
a
ping-pong
type
of
benchmark
over
over
the
network
Ethernet,
so
we've
got
producer,
pods,
consumer
pods
and
the
solar
flare
device.
Plugin
and
you'll
note
here
that
down
on
the
bottom,
our
pods
are
actually
on
the
host
network,
and
that
requires
the
pods
to
be
privileged
and
there's
a
couple
of
there's.
B
For
an
application,
but
to
be
fair,
these
pods
when
they
run
on
bare
metal
outside
of
open
ship
they're
already
running
as
a
root.
So
it's
not
really
a
regression
in
terms
of
security,
it's
just
not
as
good
as
we'd
hope
to
provide,
but
that's
why
we're
working
through
the
network
plumbing
working
group
to
make
it
possible
to
to
represent
data
networks
in
kubernetes
so
that
we
can
say
in
this
pod
in
addition
to
having
a
solar
flare,
Nick
I
also
want
to
be
on.
B
You
know
this
this
network,
let's
say
I,
have
networks,
yellow
blue
and
red
I
want
to
be
on
the
blue
network,
for
example,
and
it
will
find
a
node
that
matches
all
of
those
constraints
and
watch
the
pod
there.
So
there's
some
again
there's
a
roadmap
here,
but
this
is
how
the
the
benchmark
deployment
and
some
of
the
results
couple
of
like
world-beating
numbers
here,
we
did
have
some
overclock
Supermicro
servers
and
so
that
the
CPUs
were
running
I,
think
at
4.1
or
4.2
gigahertz.
B
The
memory
was,
as
you
know,
as
fast
as
they
sell
these
days
and
and
we
tuned
the
systems
like
crazy.
So
we
were
able
to
find
we
were
able
to
get
some
some
new
world
records
for
this
particular
benchmark
and
all
you're,
seeing
on
the
graph
here
on
the
right
is
some
statistical
analysis
of
of
the
data
itself.
B
So,
for
example,
the
mean
latency
here
between
two
machines
for
certain
packet
size
was
2.3
microseconds
and
that's
actually
the
one
that
was
that
was
the
world
record
there
and
that's
a
hundred
thousand
messages
per
second
so
and
then
the
max
latency.
There
was
just
just
10
microseconds,
which
is
which
is
great
for
software,
that
there
are
faster
solutions
which
are
kind
of
FPGA
based
generally,
but
for
software
and
for
you
know,
user
space
networking.
This
is
current
state
of
the
art.
B
So
that's
the
bare
metal
scenario
and
if
you
can
just
keep
some
of
those
maximums
in
your
head,
you'll
understand
some
of
some
of
what
I
said
earlier
around.
You
know
how
we
have
to
handle
aysil
CPUs,
so
the
benchmark
or
OpenShift
is
presented
here
and
what
I,
what
I've
trimmed
off
at
this
graph
is
actually
the
maximums.
This
were
somewhere
in
the
30
microseconds,
and
the
Delta
between
them
ends
up
being
some
jitter
introduced
by
the
way.
B
The
cubelet
currently
coordinates
process
scheduling,
so
we
do
have
excellent
numbers
somewhere
in
the
2.4,
so
essentially
2.4
or
2.3
microseconds.
So
essentially
exactly
what
bare-metal
gave
us
for
the
mean
Layton
sees,
but
once
you
step
up
into
the
you
know
three
nines
four
nines,
five
nines
and
maximums
you'll
start
seeing
outliers
and
those
outliers
are
where
our
future
activity
is
going
to
be
focused.
Getting
rid
of
those.
D
E
B
I
said
it
I
mean
when
we
go
to
these
stack
events,
it's
full
of
those
type
of
people,
so
I'll
just
quickly
describe
what
happens
when
we
and
we
will
do
FPGA
benchmarks
on
OpenShift.
It's
just
a
matter
of
finding
a
pga
vendor
that
wants
to
do
one,
because
these
are
not
cheap
but
basically
there's
two
different
types
of
banks,
there's
ones
that
are
big
enough
to
employ
their
own
barrel,
log,
designers
and
so
forth,
and
there's
ones
that
want
something
off
the
shelf
and
so
for
off-the-shelf.
B
E
B
E
B
E
B
Yeah,
so
they
implement
whatever
they
can
in
FPGA
and
there's
there's
limitations
around
how
much
can
be
done
in
Hardware
did
the
memory
limit,
so
those
devices
go
forth,
but
so
anyway,
the
whole
ecosystem
has
spun
off
around
serving
that
need,
because
it's
difficult
to
find
developers
who
are
qualified
in
so
many
different
areas
to
to
be
able
to
develop
those.
You
know
that
logic
I.
A
B
A
B
Well,
I
was
looking
for
if
there
are
any
other
questions,
I
can
get.
I
can
minimize
this
presentation
now
and
show
you
some
demos
here
before
I.
Do
that
I'll
mention
that
we
also
were
involved
with
a
bench
for
a
different
benchmark,
not
a
networking
one.
This
was
risk
analytics
on
NVIDIA
GPUs.
We
did
this
one
in
the
fall
to
prove
out
last
fall
gosh
almost
a
year
ago,
already
nine
months
to
prove
out
the
fact.
Some
of
the
new
upstream
work,
which
was
all
alpha.
B
None
of
this
stuff,
was
supported
even
in
even
in
upstream
kubernetes,
and
so
it's
taken
a
you
know
that
length
of
time
to
productize
all
of
these
features-
and
you
know,
lift
them
up
to
the
GA
level,
some
of
the
blog's
linked
there
at
the
bottom.
We
use.
You
know
at
that
time
those
Volta
Nvidia
Volta
cars
were
brand
new
impossible
to
find.
B
B
B
Okay,
sorry
about
that,
so
what
what
we're
showing
here
is
the
ability
to
do
static,
CPU
assignment.
What
we
have
to
do
is
first
change.
The
node
configuration
to
do
you
activate,
what's
called
the
static
CPU
manager
policy
and
he's
going
into
the
config
map
for
the
node
itself?
Incidentally,
and
I'll
pause
it
here,
because
it
bears
this
bares
a
little
bit
of
a
deep
dive
is
in
OpenShift
310.
The
configuration
for
a
node
has
moved
into
a
config
map.
It
used
to
be
in
a
flat
file
somewhere
in
at
sea
origin.
B
B
So
if
we
change
that
configuration
very
sorry
if
we
look
at
a
different
pod,
the
PHP
pod
that
was
running
at
this
point,
you
can
see
oops.
It
went
too
fast,
but
there's
a
there
was
a
quality
of
service
here
as
well.
That
was
on
there
and
if
you
use
the,
if
you,
if
you
use
the
CPU
manager
to
allocate
pods
and
your
pods
eyetality
cores
in
your
pods
back,
you
automatically
get
into
what's
called
the
guaranteed
quality
of
service
here.
B
So
there's
guaranteed
best
effort
and
burst
of
all
quality
of
service,
tears
and
kubernetes,
and
so
the
last
thing
in
this
demo
is,
if
we
look
in
this
PHP
pod,
which
already
had
its
pod
spec,
updated
to
request
one
core,
you
can
see
this
one
has
in
its
CPU
set.
It
only
has
one
core
rather
than
zero
and
one
so
by
default-
and
this
is
the
important
thing
to
know
about
some
of
the
future
work
we're
doing
by
default.
Any
pod
will
have
access
to
all
the
cores
and
that's
ok
for
most
workloads.
B
It's
not
OK
for
performance
workloads,
therefore,
for
performance,
sensitive
workloads
like
the
stack
benchmark
or
like
real
world.
You
know
financial
services,
workloads
or
telecom
workloads
and
so
forth.
Machine
learning,
and
so
what
we
wanted
to
do
was
create
a
way
to
select
individual
or
to
select
dedicated
cores
what
this
has
done
in
the
background,
not
part
of
the
demo,
but
it
has
pulled
that
that
one
that
number
one
core
out
of
the
global
set
of
cores
that
kubernetes
the
cube.
Let
will
schedule
two.
B
So
if
you
look
at
any
other
core
sorry,
if
you
look
at
a
CPU
set
in
in
the
Redis
pod
I
apologize,
don't
think
this
is
in
the
demo.
It
will
no
longer
say
0
1,
it
will
say
0,
because
the
core
number
1
has
now
been
exclusively
allocated
for
the
this
PHP
pod.
That's
what
this
static
state,
CPU
static,
DP
manager
policy
will
pull
certain.
You
know
whole
cores
out
of
the
total
set
in
the
system
and
allocate
them
for
certain
more
clothes.
B
What
we're
gonna
do
is
is
teach
this
CPU
manager
to
obey
the
kernel
command
line,
ISIL
CPUs
option.
So
if
you
normally,
what
happens
with
these
workloads
is
you'll?
Have
you
know
a
handful
of
course
set
aside
for
the
housekeeping
on
the
on
the
host
and
then
you'll
have
a
set
of
isolated
cores
where
the
applications
run
and
right
now
it
doesn't
coordinate
with
the
ISIL
CPUs
option,
though
it
actually
schedules.
You
know
work,
including
itself
and
heating,
the
including
the
cubelet
itself,
due
to
any
of
any
of
the
cores
that
are
isolated.
B
B
Okay,
let's
go
to
the
next
one.
Let
me
see
here
did
I
get
it
right
doing
them
out
of
order
here
may
have
closed
that
other
window.
Okay,
here
we
go
so
this
one
is
on
huge
pages
and
if
you're
not
familiar
huge
pages,
are
a
pretty
common,
optimization
technique
for
large
memory
consuming
applications,
it
helps
reduce
TLB
transaction
lookaside,
buffer
misses
in
the
kernel
and
the
CPU
and
thereby
accelerate
memory
memory
access,
so
virtual
memory
address
lookup
is
specifically
what
is
accelerated
by
these.
What
it
does
is.
B
It
increases
the
size
of
a
memory
page
in
Linux,
from
512
to
to
2
megabytes
512
bytes,
to
200
right
to
megabytes.
We
also
have
support
for
one
gig
huge
pages
when
on
processors
that
have
that
support,
so
more
modern
processors
support
two
different
sizes
of
huge
pages
and
I
should
say
I'm
talking
about
x86
here,
okay
still
I'll
hit
play
on
this
demo
and.
B
B
The
note
itself,
you
can
see
sorry
I'm,
just
hitting
pause
here
so
I
couldn't
show
yourself.
So
if
you
look
at
the
capacity
and
allocatable
fields
on
the
left
here,
I
don't
know
if
you
can
see
my
mouse
but
I'm
circling
around
them.
You've
got
huge
pages,
one
gig
and
huge
pages
to
Meg,
and
you
can
see
there
that
there
are
zero
available
on
the
system
and
that's
because
they
haven't
been
allocated
at
the
operating
system
level.
B
B
B
B
B
Try
and
sneak
in
here
and
hit
pause
yeah,
so
you
can
see
that
there's
one
gigabyte
of
tumeric
huge
pages
available
now.
So
here's
what
this
is
I
mentioned
before
that
we
have
a
new
feature
called
extended
resources.
This
was
added
by
Intel
I,
want
to
say
in
one
six,
four
one:
seven,
some
timeframe
like
that
huge
pages
and
the
device
plug-in
all
leverage
extended
resources
which
are
a
way
to
teach
the
cubelet
about
a
new
widget
about
a
new
thing.
In
this
scenario,
it's
huge
pages
and
in
the
case
of
swamp,
goes
back
benchmark.
B
He
created
a
device
plugin
that
taught
the
cubelet
aroused
solar
flare
devices.
Nvidia
has
another
plugin
that
teaches
the
cubelet
about
nvidia
hardware
and
so
forth.
That
Mellanox
has
some
prototypes,
and
incidentally,
the
Qbert
team
has
built
several
device
plugins
to
that
they're
gonna
use
in
the
Qbert
project,
which
is
how
to
how
to
run
full
on
virtual
machines
on
top
of
kubernetes
they're
using
the
device
plug-in
feature
to
do
that
anyway.
Here
you
see
one
gig
of
huge
pages
available
from
a
scheduling
standpoint.
B
So
here's
the
thing
when
you
go
through
and
have
a
pod
that
asks
now
for
X
number
of
huge
pages.
The
number
of
nodes
that
are
targets
to
be
it
to
have
it
be
scheduled
to
are
wrong,
because
only
certain
police
Urton
nodes
have
these
pages
allocated
and
presented
to
the
scheduler
through
this
capacity
field.
E
B
Hurt
there
are
perf
counters
for
the
TLB
if
you're
a
so
diana
is
asking
about.
How
do
you
measure
the
improvements
of
using
huge
pages?
Really,
what
I
just
showed
is
the
fact
that
you
can
use
them
some
more
of
a
functional
benchmark,
ax
actual
performance
improvements
and
quantifying
them.
You
know
you
can
black
box
it
and
just
you
know,
make
the
change
and
see
what
the
performance
of
your
benchmark
was
before
and
after
you're
looking
to
quantify
the
amount
of
TLB
you
know
misses
their
face.
B
E
B
B
C
B
I'm
closing
these
as
I
show
them
to
you.
The
last
two
are
last
three
demos
are
about
GPUs,
which
is
not
necessarily
the
topic
that
you
might
have
come
here
for
today,
but
it's
all
part
and
parcel
of
the
same.
You
know
bit
of
effort
and
ecosystem
upstream
I'll
work
through
those,
but
if
they're
not
in
your
interest
area,
go
ahead
and
feel
free
to
drop
their
their
short
would
be
done
in
the
next
ten
minutes.
I
would
say,
but
also
feel
free
to
ask
questions
whenever
you
have
them.
There's.
B
Yeah
each
individual
feature
has
only
been
tested
up
to
well.
I
would
say
this
huge
penis
has
been
tested
at
up
to
2,000
notes.
We
don't
have
an
environment
that
has
2,000
nodes
that
have
GPUs
in
them,
so
that
has
not
been
done.
However,
Nvidia
themselves
told
me
they
tested
up
to
650
notes
with
GPUs.
Hopefully
that
answers
your
question.
B
B
So,
let's
get
started
here,
someone
better
than
me
at
graphics.
You
know
with
this
video
a
little
I
should
mention
this.
This
demo
itself
was
was
done
by
one
of
our
was
done
by
me,
but
the
actual
content
of
it
was
created
by
funko,
and
we
he
blogged
about
it
in
open
ship,
3,
9
timeframe,
I
think
feel
free
to
jump
in
here
as
well.
Go
if
you
can
hear
us
what
you're
looking
at
here
is.
B
If
you
remember
before
we
had
the
limits
still
declare
here.
If
I
hope
you
can
see
this,
but
the
limits
and
video.com
last
GPU.
The
important
piece
here
is
that
it's
we're
requesting
two
GPUs,
so
it
will
only
land
on
nodes
that
have
two
GPUs
in
them,
and
we're
gonna
show
eventually
the
difference
in
performance
of
image,
recognition
benchmark
when
we
run
it
on
one
GPU
versus
when
we
run
it
on
two
GPUs
and
it
doesn't
necessarily
have
anything
to
do
with
openshift
itself.
But
just
the
fact
that
you
can
get
this
all
running
on.
B
So
we've
got
a
pod,
that's
got
cafe
to
running
and
a
jupiter
notebook
in
it-
and
this
is
this-
is
one
of
the
tutorials
that
ships
with
cafe.
Let
me
pause
it
here.
You
can
see
that
one.
This
is
actually
just
one
GPU
running
and
you
can
see
that
it's
capable
of
analyzing,
roughly
15
758
images
per
second,
so
there's
only
one
GPU
being
used
in
this
jupiter
notebook
in
the
backend.
B
Further
down
we'll
see
that
I
must
pause
it
here
again,
we
added
a
second
GPU,
so
this
part
of
the
jupiter
notebook
requests
or
actually
implements
multiple
GPUs
and
again
these
these
tutorials
are
part
of
cafe.
So
you
can.
You
can
use
these
yourself,
and
this
capable
is
essentially
linearly
scaling
here.
B
115
images
per
second
now
across
two
GPUs
and
one
thing
I'll
say,
is
that
I
wouldn't
keep
those
those
absolute
numbers
in
mind
if
you've
ever
run
any
of
these
benchmarks,
you
know
and
video
talks
about
being
able
to
do
a
thousand
on
one
GPU
in
their
newest
gear,
so
these
are
some
of
the
lower
end
in
GPUs
available
on
Amazon.
But
again
it's
just
proving
out
the
functionality
here,
you're,
not
necessarily
the
robber.
B
E
A
B
B
B
B
Okay,
so
the
next
one
is
or
another
chat,
okay
yeah,
you
can
have
copies
of
all
this
stuff.
Okay,
so
this
one
is
around
sis
controls
and
this
one
is
not
gatech
preview
in
310
I,
don't
know
if
it
made
111
upstream
I
think
it's
still
gonna
be
beta,
which
may
which,
which
mean
maybe
maybe
makes
maybe
we
can
graduate
this
thing
to
GA
in
311,
but
I
actually
have
to
follow
up
on
that
I
don't
know
so.
B
This
is
just
a
minute
long
here,
just
a
quick
overview,
so
certain
applications
require
dis
controls
to
be
tuned
and
one
of
those
a
good
example
of
that
is
elasticsearch
it.
The
latest
versions
of
elasticsearch
require
certain
virtual
memory
and
I
forget
the
exact
sysctl,
but
it
won't
even
start
unless
you've
increased
this
certain
virtual
memory
tunable.
So
that's
a
concrete
example
of
an
application
that
will
not
run
on
kubernetes
without
being
able
to
tweak
these
things.
B
Now,
when
you,
whether
you
do
it
at
the
host
level
or
you
do
it
inside
a
container
are
two
different
things:
it
needs
it
tuned
at
the
host
level
and
OpenShift
310
cluster
I'm,
just
showing
here
quickly
that
that
of
certain
this,
this
certain
assist
control
is
set
to
one
there.
Sorry,
it's
set
to
zero
by
default
in
rel.
B
Here's
my
template
and
I'll
quickly
pause
it
here.
The
the
reason
that
this
thing
is
not
beta
yet
is
because
you
see
the
alpha
implementation
uses
annotation
and
the
process
of
converting.
You
know
the
initial
alpha
implementation
over
to
data
requires
it
to
move
out
of
annotations
into
a
legitimate
field
into
a
real
field
within
the
pod
spec,
and
that's
the
process
that
was
done
by
our
team
over
the
last
couple
of
months.
B
You
can
tweak
at
a
per
pod
basis,
but
you
have
no
like
contract
with
the
rest
of
the
community
that
these
have
been
validated
as
being
safe
from
a
kernel
standpoint
and
you
actually
need
to
reconfigure
there's
a
flag.
You
need
to
pass
to
the
cubelet
to
allow
unsaved
sis
controls
and
you
actually
have
to
provide
an
array
of
the
ones
that
you
want
to
whitelist
at
the
cubelet
level,
to
allow
pods
to
then
change
those
sis
controls
and
then
the
third
level
are
the
node
level.
B
Anything
to
do
with
the
network,
I
should
say
some
of
the
network.
Buffer
sizes
are
global
or
sorry.
Our
per
interface,
I
should
say
and
not
namespace
aware
most
of
the
memory
management
subsystem
is
not
namespace
aware
so
there's
those
three
classes
safe,
unsafe
and
node
level,
though
that
answer
your
question.
C
B
Yeah,
there's
there's
reasonable
documentation
on
OpenShift
or
or
kubernetes
site
about
what
I
just
described.
Unfortunately,
this
is
an
area
that
hasn't
had
particularly
the
triage
of
what's
what's
falls
into
the
safe
and
unsafe
sis
control
area
is
really
just
about
prioritization
like
if
you
find
assist
control.
We've
done
this
on
behalf
of
customers.
They
sent
us
a
list
of
five
sis
controls.
Red
Hat
went
through
and
verified
that
those
were
safe
and
then
made
it
possible
to
tune
those
upstream.
B
So
it's
really
just
a
matter
of
prioritizing
and
if
it's
not
possible
to
do
them
to
make
them
namespace
aware,
then
we
have
a
lot
of
work
in
front
of
us
to
make
them
namespace
aware
upstream
kernel
work
which
may
or
may
not
even
be
technically
possible
or
even
something
the
kernel
community
is
interested
in
entertaining
so
to
handle
things
like
that.
We
have
the
node
level,
sis
controls
and
some
of
these
digital
er,
like
intelligent
routing,
scheduler
decisions
that
I
was
trying
to
get
across
earlier
through
the
extended
resources
stuff.
B
B
So
what
we
did
was
we,
we
tainted
those
nodes,
pretty
sure
he
tainted
those
nodes,
which
means
that
you
have
to
have
a
corresponding
toleration
on
the
pod
itself
in
order
to
get
scheduled
to
those
nodes.
So
essentially,
nothing
runs
there
unless
there's
a
special
flag
in
the
in
the
pod
spec
to
protect
those
precious
resources
and
protect
those
nodes
from
having
you
know,
just
a
random
WordPress
running
on
them.
C
B
Yeah,
excuse
me:
yes,
if
that
is
true.
Okay,
so
I'm
gonna
hit
play
on
the
there's.
Only
a
couple
minutes
left
this
demo
is
kind
of
funky.
It
shows
you
how
to
create
your
own
resource
and
let's
say
you
have
a
resource,
that's
unique
to
your
environment.
That
kubernetes
doesn't
know
about
yet,
and
you
want
a
schedule
against
it
in
this
case,
I'm
talking
about
for
the
demo
purposes,
the
concept
of
a
dongle,
some
some
applications
have
a
physical
dongle
that
comes
with
them.
B
B
I've
got
a
little
script
here
that
adds
or
it
updates
the
capacity
and
allocatable
fields
of
a
node
to
provide
or
to
publish
the
availability
of
42
dongles.
So
this
note
magically
set
thinks
it
has
42
dollars
on
it
now
and
if
I
get
the
allocatable
and
pass
a
fields,
you
can
see
here
that
example.com
/
dongle
goes.
B
Cool
so
yeah,
so
now,
if
I
get
the
list
of
pods,
you
can
see.
You
know
now
that
a
node
showed
up
that
that
could
supply
the
three
dongles.
Now
it
is
now
kubernetes
has
scheduled
the
pod
to
that
node
and
started
the
start
at
the
pot,
though
that
I
think
is
it
for
the
demos
for
today
guys
I,
guess
the
end
here.
Just
deleting
that
resource,
just
deleting
those-
and
you
can
see
here
in
the
capacity
people
table,
there's
no
more,
no
more
dongles
in
this
cluster.
That's.
C
D
B
B
We
used
a
profile
called
CPU
partitioning
which
creates
the
group
of
housekeeping
cores
versus
degenerate,
isolated
cores
and
so
we're
trying
to
really
make
it
super
easy.
The
tuna
system
for
a
telecom
or
financial
services
workload
with
that
profile,
all
you
have
to
do
is
tell
it
which
cores
are
housekeeping
and
which
cores
are
isolated
and
reboot
the
node,
and
suddenly
you
have
an
environment,
that's
tuned
for
that
type
of
workload.
B
B
Have
some
proof
of
concepts
underway
I'm,
not
able
to
mention
names,
but
we
do
have
some
proof
or
concepts
under
way
for
all
of
this
stuff,
and
certainly
there
are
some
customers.
You
know,
even
though
they're
not
graduated
to
GA
there.
There
are
quite
a
few
customers
running
GPUs
on
OpenShift
in
you
know,
don't
quite
a
few
in
production.
B
A
B
B
A
A
Maybe
you
want
to
one
or
two
of
the
POC
folks
to
talk
us
through
their
lessons
learned
in
the
use
cases,
because
there's
lots
of
different
use
case
as
to
which
this
work
applies.
Not
just
trading
platforms,
though
it's
very
useful
information
and
thank
you
again,
Jeremy
for
taking
the
time
to
talk
with
us
today.