►
From YouTube: OpenShift Commons OpenShift on OpenStack SIG #1: Deploying a 1000-Node Cluster on the CNCF Lab
Description
This week’s OpenShift on OpenStack’s guest speaker is Jeremy Eder from Red Hat’s Performance and Tuning team who will be discussing the findings from his work deploying both OpenStack and OpenShift on CNCF.io’s Cluster. He’ll also be discussing the test harness that used to do the scaling tests and the lessons learned.
Here’s a link to the blog he wrote, https://cncf.io/news/blogs/2016/08/deploying-1000-nodes-openshift-cncf-cluster-part-1
From the blog:
'Imagine being able to stand up thousands of tenants with thousands of apps, running thousands of Docker-formatted container images and routes, on a self-healing cluster. Take that one step further with all those images being updatable through a single upload to the registry, all without downtime. We did just that on Red Hat OpenShift Container Platform running on Red Hat OpenStack on a 1000 node cluster.'
A
A
Well,
hello
and
everyone
welcome
to
this
sort
of
second
open
shift
on
OpenStack
sig
meeting
the
first
one
was
part
of
a
briefing
and
Jed
Matlin
was
the
one
who
spoke
at
that
one.
He
gave
us
a
wonderful
overview
of
the
reference
architecture
from
dell
openshift
on
OpenStack
and
we
thought
because
it
was
such
an
interest
in
that
we
would
definitely
try
and
and
host
sort
of
bimonthly
opens
open
shift
on
OpenStack
cigs,
and
today
we
have
Jed
with
us
and
Jeremy
eater.
He
wrote
a
wonderful
blog
post
about
his
work.
A
Testing
Oh
be
shipped
on
OpenStack
on
the
CNC
f
lab,
which
is
the
cloud
native
computing
foundation,
and
the
MTF
has
a
cluster,
which
is
a
wonderful
thing
that
was
donated
by
Intel
and
is
I
think
it's
super
nap
I'll
switch
one
of
those
places
and
it's
been
a
great
test
bed
for
us
for
doing
some
scaling
testing.
So
we
thought
that
would
be
a
good
way
to
open
up
the
conversation.
Since
this
is
a
cig
and
not
a
briefing
of
sorts
and
jeremy's
is
kindly
I'm
often
to
do
this
talk.
A
We
do
have
a
chat
for
this.
So
raise
your
hand
in
the
chat.
If
you
want
to
ask
question,
I'm
going
to
unmute
everybody,
you
can
sell
commute
if
you
have
dogs
and
cats
and
kids
and
heavy
equipment
in
the
back
room,
but
then
just
raise
your
hand
and
ask
a
question
and
then
at
the
end
of
jeremy's,
will
open
it
up
for
a
conversation
about
what
this
cig
means
to
you
and
where
we
want
to
go
with
it
all
right.
So
I'm
going
to
turn
let
Jed
introduced
himself
and
then
we'll
kick
it
off.
A
B
We
have
the
gathering
coming
up
right
before
khoob
con
in
Seattle
in
early
November,
looking
forward
to
seeing
a
whole
bunch
of
folks
there
and
learning
and
teaching
tons
about
what
we're
doing
the
my
work
is
available
through.
Unfortunately,
your
dell
sales,
rep
and
I'm
also
very
proud
and
surprised
that
intel
bought
the
thousand
nodes
of
adele
key
or
del
r
6
30
s
which
are
featured
in
my
reference
architecture
for
open
shifts.
So
we
can
actually
go
pretty
deep
into
the
configuration
of
these
boxes.
C
Yeah
hi
hi
everybody.
My
name
is
Jeremy
dhuryam
engineer
in
the
performance
and
scale
engineering
team
at
Red,
Hat
and
these
past
couple
years
been
working
on
openshift
container
technology
in
general
and
in
the
month
of
june
or
may
we
became
aware
of
the
cnc
f
environment
that
it
was
going
to
be
become
available
and
we
had
a
product
need
to
get
some
additional
scale.
C
Testing
done,
and
here
we
are
so
we
were
able
to
use
that
lab
and
go
over
some
of
the
details
about
how
we
used
in
why
we
did
things
in
a
certain
way
and
and
some
of
the
results
you
know
which
are
which
are
I,
guess
the
most
interesting
part
and
for
me
I
guess
the
journey
was
equally
interesting.
A
lot
of
work
came
out
of
it
and
there's
more
work
to
go
but
and
we're
actually
right
now
gearing
up
for
an
additional
scale
test.
C
C
Okay,
what
I
guess
I
I
figured?
Maybe
it
would
be
interesting
to
take
you
guys
through
how
we
turn
requirements
into
results
from
our
perspective
and
that
will
start
with
the
lofty
kind
of
marching
orders
from
the
product
managers.
Saying
here's
what
we
need
to
do
or
here's
what
we'd
like
you
to
try
and
get
to,
and
you
tell
us
how
realistic
this
is
and
go.
Try
it
anyway.
C
How
C
ncf
got
involved
I
want
to
share
with
you
the
deployment
of
the
open
shift
on
OpenStack
at
scale
that
we
did,
which
we
learned
a
lot
from
and
is
feeding
into
our
product
documentation
and
our
own
reference
architectures
along
these
lines.
I
also
wanted
to
take
you
into
some
of
the
details
about
how
how
we
actually
do
our
testing,
because
the
results
and
the
numbers
have
less
have
more
meaning
that
way.
If
you
understand
what
how
we
are
actually
exercising
the
environment,
some
results.
These
are
the
ones
that
are
shared
in
the
blog.
C
So
if
you've
read
the
blog
already,
you'll
have
seen
some
of
these
results
and
then
hopefully,
some
QA
and
as
I'm
going
through
this
I
didn't
prepare
a
terrible
amount.
You
know,
but
so
so
please
do
feel
free
to
ask
questions.
Diane
I
cannot
see
the
chat
while
I'm
looking
at
the
slide.
So
just
you
know
interrupt
me
if
there's
questions
and
oh,
if
I'm,
in
going
off
in
the
wrong
direction,
just
refocus
me
here:
okay,
so
what
did
we
set
out
to
do?
C
We
get
a
product
requirements
document
from
the
product
managers
in
a
release,
planning
phase
in
advance
of
code
freeze
for
open
shift,
33,
which
I
guess
was
a
couple
of
months
ago
now,
and
our
goals
were
pretty
lofty
along
the
way
we've
been
getting
more
and
more
information
and
feedback
from
the
field.
That
density
and
consolidation
is
a
very
important
attribute
for
the
system.
So
to
that
end
we
decided
to
try
and
up
the
number
of
pods
that
can
be
supported.
C
Per
node
openshift
provides
an
elastic
search,
fluent
d
and
Cabana
based
stack
for
consolidated
logging.
We
wanted
to
exercise
the
added
as
biggest
scale
as
we
can.
It
also
provides
a
hawk
Uhler
heap
stir
and
cassandra
based
metrics,
a
stack
that
runs
again,
although
actually
logging
in
metrics
all
run
as
pods
on
top
of
open
shift.
So
we
wanted
to
test
that
and
then
we
also
had
a
like
a
full
density
kind
of
it's
more
of
a
control
plane
exercise
at
full
scale
at
as
dense.
We
as
we
could
get.
So.
C
Let
me
just
say
one
thing
before
I
continue
the
support
statements
that
come
out
of
open
shift
and
out
of
Coober
Nettie's
are
generally
around
how
many
pods
you
can
run
and
how
many
pods
you
can
run
at
a
certain
number
of
nodes.
So
it's
it's
a
little
bit
difficult
to
grok,
but
the
the
idea
is
that
fewer
pods
you
can
have
fewer
pods
on
on
nodes.
The
more
nodes
you
have
so
at
full
scale
thousand
nodes
were
not
being
we're
not
doing
250
pods.
C
We
were
doing
half
of
that,
and
so,
if
you
needed
to
get
to
250
pods
on
a
node,
you
would
only
be
able
to
have
500
notes.
In
other
words,
125,000
pods,
apparently,
is
the
maximum
for
open
shift
33.
So
the
idea
was
okay,
given
these
type
of
marketing
marching
orders,
what
what?
How
the
heck
are
we
going
to
do
this?
How?
Where
are
we
going
to
do
it?
How
are
we
going
to
do
it
and
we
already
have
the
why
so
from
there?
C
We
we
have
previous
relationship
with
Chris
dansak
from
Sen,
CF
and
and
Dan
cone.
So
we
discussed
our
use
of
their
lab
extensively
and
what
ended
up
happening
was
we
were
allocated
300
of
those
physical
notes,
so
everyone
was
bursting
through
doors
and
windows.
Like
the
kool-aid
man
saying:
okay,
we
got
this
kickass
gear,
it's
super
high-end
stuff.
We
really
can't
even
get
higher
end
stuff,
even
if
we
had
infinite
budget,
because
you
know
every
system
has
an
nbme
device.
C
For
example,
there's
just
fourteen
thousand
plus
cores
that
were
available
here,
really
really
awesome
gear,
Dell
gear
and
plenty
of
RAM.
Basically,
we
wanted
for
nothing
in
terms
of
infrastructure.
Here
ten
well
I
will
say
this:
the
10
gig
next
week
we
probably
could
have
used
a
few
more
of
those
but
anyway,
so
it
that's.
The
hardware
stack
dell
poweredge
r6
30s,
with
with
Haswell
chips
and
256
gigs
of
ram,
so
really
great
gear
for
being
able
to
do
the
try
and
hit
the
targets
that
we
are
trying
to
get
to.
C
We
were
able
to
focus
ourselves
on
the
software,
which
was
which
is
really
untying
our
hands
in
a
lot
of
cases
when
we
try
and
do
scale
testing,
sometimes
its
a
mix
of
well,
we
want
a
partner
to
do
that
or
well.
We
don't
have
the
software
to
do
that.
In
this
scenario,
the
hardware
question
was
answered
for
us,
thankfully,
by
CN
CF.
So
here
we
are
that's
the
left-hand
side
of
the
page
right
bottom
quadrant
talks
about
the.
B
Birth
before
we
get
to
the
software
I'm
curious.
If
you
know
what
sort
of
northbound
networking
gear
is
involved
for
our
dell
RA,
we
have
a
full
cross-linked
H,
a
switch
infrastructure
both
for
northbound
and
east
west
bound
across
racks.
Our
solution
in
our
ra
goes
up
to
three
full
racks.
This
is
a
lot
more
than
three
full
Rex,
but
you
could
still
get
a
lot
of
processing
car.
So
what
what
do
you
know?
Who
is
supporting
supplying
the
networking
gear.
C
You
know
the
lab
was
turned
over
to
us
as
machines,
so
I
don't
know.
I
know
we
had
problems
at
the
network
switch
level
with
dropped
packets
that
we
were
working
on
for
a
while
that
ended
up
getting
resolved.
He
also
had
driver
bugs
in
the
Intel
I
for
TE
driver
that
made
it
look
like
there
might
be
switching
problems
when
there
weren't
so
I,
don't
actually
know
what
the
uplink
switch
models
were.
We
they
were
believed
they
were
Cisco,
but
I
could
be
wrong,
though,.
C
C
That
would
allow
that,
in
a
way
that
their
Ops
guys
over
there
work
very
comfortable
with
so
one
of
the
ideas
was
to
dump
the
switch
configs
for
us
every
day
and-
and
you
know,
process
them
such
that
they
only
see
what
we're,
what
we're
supposed
to
see,
or
even
do
it
more
often
than
that,
so
we
could
have
at
least
a
text-based
output.
We
were
looking
for.
Things
like
you
know,
is
the
switch
dropping
packets
do
we
have
linked?
We
actually
had
a
bunch
of
link
errors.
C
You
know
just
driver
bugs
on
you
know
that
were
related
to
even
establishing
link
with
the
switches,
so
those
are
things
that
were
difficult
to
debug
without
access
to
the
switching
infrastructure,
and
this
is
just
the
nature
of
the
way
this
lab
was
given
to
us
at
the
time.
So
part
of
our
feedback
to
them
has
been
we
would
like
read-only
access
or
something
similar
around
the
switching
as
well.
As
you
know,
we
need
we
would
love
to
have
like
net
flow
type
data
from
the
switching
to
understand.
C
If
we're
hitting
a
cross
connect,
link
contention-
or
you
know,
bandwidth
issues
along
those
lines
that
we
never
did
stress
the
control
the
networking
plane
to
the
point.
Where
were
pushing
10
gigs
of
traffic,
but
from
a
functional,
functional
standpoint,
it
might
have
speeded
up
our
debugging.
That's
very
lab
specific.
It
really
didn't
have
much
to
do
with
open
shift
and
OpenStack,
but
you
know,
since
you
asked.
C
Okay,
so
we
did
deploy
with
rell
72.
We
had
to
upgrade
to
relabel
73
development
kernel
in
order
to
pick
up
the
driver
fixes
that
I
had
mentioned
earlier,
the
bugs
that
were
fixed
upstream,
that
are,
that
weren't
fixed
at
the
code
level.
We
had
at
the
time
their
cart
they're
fixed
at
this
point
and
then
OpenStack
rented
OpenStack
platform
8,
which
is
based
on
Liberty
you're,
familiar
with
their
naming
scheme,
a
bunch
of
patches
that
are
again
publicly
released
arata
and
we
used
open
shift
33,
which
was
an
alpha
alpha
stage.
C
At
the
time
these
tests
were
run.
So
that's
the
mix
of
hardware
and
software
resources
that
we
used
now
poop
up
for
private
cloud.
This
is
probably
a
fairly
standard
config.
Maybe
it's
a
little
bit
larger.
Maybe
the
nodes
are
a
little
bit
nicer
than
many
people
can
afford,
but
galing
those
down
you
know
I
would
think
this
is
a
pretty
happy
path
for
for
customers
who
are
doing
private
cloud.
At
this
point,
they're
11
there
were
a
lot
of
things
that
were
more
nice
to
have
than
need
to
have
in
this
environment.
C
C
So
the
logical
diagram
that
we
built
is
on
the
right
and
I.
Hopefully
you
can
see
that
there
were
that
the
colors
are
blue
as
a
physical,
node
and
green
as
a
virtual
mode,
and
what
we
did
was
we
created
two
different
host
aggregates,
which
are
essentially
like
regions
or
zones
and
in
each
one
of
those
regions.
C
So
that's
just
another
vm
and
we
did
end
up
with
only
three
diagram
says
that
it
goes
up
to
o
master
and
there
was
actually
three
masters
and
three
stds,
and
then
there
was
a
support
of
availability
zone
or
host
aggregate,
and
in
there
we
stuck
the
routers
registries
metrics
and
logging
pods
that
were
in
use.
So
the
third
part
was
a
catch-all
which
was
everything
else.
In
our
case
it
ended
up
being
just
nodes
and
it
was
a
fourth
actually
where
we
stuck
a
bunch
of
servers.
C
We
had
in
reserve
where
we
ran
our
workload.
Generators
from
and
those
are
just
bare
metal,
realm
nodes
that
and
our
workload
generator
currently
is
this
jmeter,
so
it
generates
web
traffic.
Some
of
the
flavors
we
used
are
at
the
bottom
of
the
diagram.
Here.
All
of
our
nodes
were
for
V
CPUs,
16,
gigs
of
RAM
and
32
gigs
disk
and
those
were
probably
larger
than
we
need
it.
C
But
it's
kind
of
the
thing
where
we
had
a
bunch
of
hardware
we
figured
we
would
use
all
of
it
and
then
the
HDD
servers
and
infra
nodes.
Those
definitely
need
to
be
beefier,
there's
a
depending
on
the
type
of
test
or
workload.
That's
going
on.
You
generally
need
more
cpus
to
handle
the
metrics
and
logging
and
the
router
and
registry,
depending
on,
if
you're
doing
web
traffic.
You
know
the
routers
would
get
busy
on
CPUs.
C
The
registries
do
a
lot
of
CPU
activity,
but
mostly
disk
activity,
and
so
the
nodes
10,
the
infra
and
FC
denotes
tend
to
be
need
a
little
bit
more
CPU
resources
and
then
finally,
the
most
the
most
cpu
intensive
ones
are
the
masters
and
those
are
the
ones
that
are
constantly
busy
doing
command
and
control
traffic
with
the
notes.
Part
of
our
effort
in
this
time
frame
was
to
improve
those
command
and
control
channels,
and
we've
done
that
a
couple
of
ways.
First
of
all,
we've
put
into
some
some
caches
and
we've
done.
C
We've
added
some
sed
washes
if
you're
not
familiar
with
those
a
watch
is
a
persistent
connection
that
a
node
will
establish
on
the
master
that
allows
it
to
or
sorry
oh
yeah,
via
the
master
that
allows
it
to
be
constantly
updated
with
status
changes
from
the
cluster.
So
a
node
will
immediately
know
whether
or
not
a
new
node
became
available,
and
it
knows
how
to
route
packets
to
it
because
it
has.
It
has
received
that
update.
C
Basically,
we
found
that
go
legs,
JSON
library
was
or
could
use
some
optimization,
and
so
the
Google
team
has
invented
protocol
buffers
which
they're
in
use
in
more
than
Cooper
Nettie's,
but
we
applying
them
to
Cooper
Nettie's
allowed
us
to
reduce
the
CPU
usage.
I
believe
from
22
cores
down
to
15
cores,
or
so
so,
maybe
thirty
or
forty
percent
improvement
in
the
number
of
cores.
It
takes
to
manage
a
thousand
notes,
a
tremendous
improvement
between
the
caches
and
the
protocol
buffers,
so
phenomenal
scalability
improvement
there.
C
So
for
a
while
to
get
to
these
numbers,
we
were
just
throwing
more
hardware
at
it,
making
that's
why
the
masters
are
for
TV
CPUs
at
a
point
we're
up
to
22.
You
know,
and
we
really
felt
like
we
didn't-
want
to
be
bottlenecked
on
CPU,
so
we
threw
as
much
hardware
as
we
could
at
it
and
at
this
point,
what
we're
doing
is
whittling
down
those
numbers
to
to
allow
the
masters
to
just
be
just
be.
C
The
overall
efficiency
in
the
control
plane
is
is
what
we're
trying
to
address,
and
in
order
to
do
all
of
that
not
only
did
we
involve
all
of
the
special
interest
groups
in
in
Coober
Nettie's
land,
like
particularly
sake
scale,
but
also
openshift,
engineering,
quality
engineering
inside
Red
Hat,
the
OpenStack
engineering
group,
our
group,
the
performance
team
and
then
analogous
to
the
teams
that
that
Judge
is
likely
on.
We
have
reference
architecture
teams
which
are
writing
documents
that
to
represent
these
best
practices
that
we
learned.
C
B
C
Yeah
the
load
balancer
is
a
service
in
OpenStack
is,
is
not
yet
actually
supported
by
Red,
Hat
and
so
what's
used
in
this
installation
is
the
native
load
balancer,
which
is
H
a
proxy
base
which
comes
with
open
shift.
Whenever
you
install
open
ship
with
more
than
one
master,
you
will
automatically
get
and
you
have
to
configure
you
know
a
load,
balancer
node.
C
C
So
we
have
the
environment.
Let's
say
we
have
that
environment
built
out.
What
are
some
of
the
things
that
we
wanted
to
test?
Our
team
is
again
we're
on
a
performance
team
and
we
collaborate
with
our
quality
assurance,
guys
too
develop
tests
for
open
shift,
and
so
our
repository
is
linked
here.
It's
called
SVT
and
I
didn
t
admit
so
I,
don't
know
why
it's
called
that,
but
other
than
the
fact
that
it
stands
for
system,
verification
and
test.
So
those
are
open
source.
You
can
grab
them
right
now
and
look
at
what
we
do.
C
There's
four
main
things
that
are
in
there,
at
least
from
my
point
of
view,
there's
actually
a
fair
amount
more,
but
the
main
and
interesting
ones
for
this
subject
matter
we're
going
over
our
cluster
loader
Network
tests,
work,
load,
generator
and
we've
got
some
reliability
test,
which
were
we
look
for
memory,
leaks
and
connections
that
don't
close
properly.
There's
a
fair
amount
of
TCP
connections,
HTTP
connections
in
the
system.
C
C
That's
got
no
configuration
on
it
in
terms
of
open
shift,
so
it'll
set
up
the
partitions
make
sure
that
the
doctor
storage
is
set
up
correctly
bakes
in
you
know,
a
lot
of
I
mean
this
tech.
This
type
of
image
gold
image
techniques
been
around
forever.
This
is
just
automating
it
through
ansible
and
wiring,
in
the
open
shift,
specific
bits
which
are
really
just
pre,
pulling
a
lot
of
the
images
that
are
required
to
run
up
and
shift.
C
So
if
you've
seen
it
before,
there's
like
0
SE,
pod,
ose,
router,
ose
itself-
and
you
know,
there's
maybe
a
dozen
of
them
that
we
need
to
pre
poll
on
to
each
node.
The
reason
we
do
that
is
because,
if
you're
doing
thousands
of
nodes
or
a
thousand
nodes
going
over,
the
network
becomes
a
point
of
contention
to
pull
all
of
that
stuff,
and
so
baking
it
into
the
images,
became
a
necessity
actually
in
order
to
get
any
done
in
the
time
frames
that
we
had.
C
So
what
we
did
is
we
push
this
up
into
github?
You
can
use
it
yourself
right
now
and
it's
you
know.
As
long
as
you
configure
it
inventory
file
for
your
environment,
it
will
spit
out
a
cute
cow
too,
or
it
will
also
spit
out.
An
Amazon
am
I
that
has
all
this
stuff
baked
in,
though
a
means
of
self-defense
really
to
get
work
done.
You
don't
necessarily
need
to
do
this,
but
it
will
take
you
a
lot
longer
if
you,
if
you
didn't,
do
something
like
this.
C
Those
are
the
primary
two
things
that
you
get
yeah
yep
and
we
did
some
stuff.
We
put
our
monitoring
solution
in
there
are
we
have
a
metric
system
and
monitoring
solution
called
P
bench.
We
wanted
to
make
that
in
what
that
does.
Is
gather
system
metrics
and
provide
graphs
for
us
at
the
system
level
so
that
we
can
understand.
You
know
when
we
run
these
tests.
What
is
the
impact
and
efficiency
I
mentioned?
You
know
dropping
from
22
to
15
cores.
That
was.
C
Okay,
so
now
we've
got
open,
stack,
open
shift
on
top
and
we've
got
our
and
we've
got
our
images
deployed
the
clusters
built
at
this.
Let's
assume
the
clusters
built
at
this
point
again.
What
do
we
do
with
it?
We
designed
some
tests
that
are
centered
around
a
shared
web
hosting
scenario,
which
is
kind
of
the
bread
and
butter
for
Cooper
Nettie's,
an
open
shift
at
this
phase
in
its
life.
We
expect
that
to
change
drastically
over
the
next
year
or
two,
but
right
now
it's
centered
around
12
factor,
application,
development
and
web
based.
C
You
know
web
languages
like
or
post
breasts
and
node
and
PHP
or
whatever
else
your
favorite
thing
is
open
shift
itself,
ships,
examples
which
are
templates
and
we've
used
those
in
the
so
them
to
our
needs.
So
what
we
also
develop,
a
utility
called
the
cluster
loader
and
it
shows
up
in
the
blog.
What
it
is
is
a
way
to
express
an
environment
that
consists
of
all
the
complexities
of
Cooper
Nettie's
like
represent,
really
what
a
customer
is
trying
to
do,
or
users
going
to
do.
C
You
know
as
if
we
had
a
ton
of
Legos
and
we
could
build
whatever
we
wanted
to.
With
these.
With
these
building
blocks,
the
config
directory
contains
and
I'll
show
to
you
in
a
slide
just
the
guts
of
what
the
cluster
10
will
go
off
and
do
and
it's
a
Python
script
right
now
and
we
have
some
plans
on
how
to
improve
it,
as
you
could
imagine
anyway.
So
that's
the
config
and
content
directories,
including
the
templates,
the
cluster
loader,
the
Python
script
itself,
which
takes
only
two
arguments.
C
You
know
what
actually
one
argument:
what
file
am
I
living
it
from
and
optionally?
Second,
arguments
around
using
the
work
load,
generator
portion
of
cholesterol
or,
and
then
of
course,
a
utilities.
That's
got
out
just
a
bunch
of
functions
in
it.
So
that's
the
framework
that
we
use
to
test
openshift.
C
The
architecture
is
pretty
simple,
probably
the
most
simple
flow
chart.
You
can
have
start
it
off
Phil
through
the
arguments,
create
a
namespace
or
a
project
and
open
shift,
our
lands
and
within
that
namespace
create.
However
many
things
you
want
to
represent
what
your
applications
might
be
doing:
quotas,
users,
pods
that
are
and
then
at
the
end,
iterate
until
you've
reached
the
total
number
of
pods
and
then
exit,
and
at
that
point
we'd
have
an
environment,
that's
ready
for
the
second
phase,
which
is
actually
putting
work
on
it.
This
is
the
first
phase
and
oftentimes.
C
This
is
the
only
phase
that
matters,
because
potentially
we
could
be
using
this
as
not
only
the
loading
utility
but
also
benchmarking.
How
quickly
do
we
load
applications
when
the
cluster
is
under
load?
What
is
the
API
response
of
slow
response
times
etc
under
load?
And,
additionally,
how
many
CPUs
is
it
take
us
to
get
this
a
certain
fixed
amount
of
work
done.
C
The
test
we
ran
inside
CNC
f
is
the
guts
of
it
are
pretty
lengthy
and
the
actual
test
is
on
the
blog,
but
here's
an
example
that
we
here's
an
example.
That's
in
our
repository
and
the
number
of
projects
is
really
all
that
changed
here.
This
is
the
left
top
you'll
see
in
numbers
only
five,
so
it
would
deploy
five
projects
each
with
one
user,
with
three
templates
inside
of
a
build
configs,
six
build
templates,
one
image
stream
to
replication
controller
templates.
C
C
C
So,
instead
of
five,
we
tried
to
go
as
high
as
we
could
until
failure,
and
this
is
roughly
what
we
did
well.
This
is
exactly
what
we
got
2,000
nodes,
13,000
projects.
Our
target
was
20,000.
We
didn't
get
there
in
the
time
allotted
52,000
pods,
you
know
I'm
not
going
to
read
the
rest
of
it,
but
numbers
are
pretty
pretty
impressive.
I
guess,
and
I
think
we
learned
a
lot.
You
know
we
learned
a
lot
about
where
to
adjust.
Config,
what
to
fix
and
the
deliverables
were.
C
There
were
bug,
fixes
a
lot
of
them
and
a
lot
of
it
actually
ended
up
being
documentation
as
well.
For
example,
if
you're
doing
logging
scalability
if
you're
doing
a
big
environment-
and
you
want
the
you
know
the
the
elastic
search
to
scale,
here's
the
three
or
four
different-
you
know
things
you
need
to
do.
For
example,
systemd
has
a
thousand
messages,
a
second
rate
limit
by
default.
C
C
C
As
much
as
it
is
around
the
number
of
images
so
for
the
next
generation
test,
you
can
imagine
we're
going
to
pile
on
the
number
of
images
and
come
up
with
the
exact
capacity
planning
formulas
around
around
what
you
can
expect
there
and,
on
the
right
hand,
side
towards
the
right
hand,
side
of
the
graph.
You
can
see
a
fairly
busy
cpus
that
suddenly
dropped
off
and
yeah.
That
was
a
bug
that
we
encountered
and
that's
actually
where
we
ended
our
test
pretty
much,
because
we
had
just
run
out
of
time.
C
We
got
that's
how
we
got
to
13,000
projects
instead
of
20,000.
That's
basically
the
point
where
things
fell
over
and
yeah.
We
had
to
bail
at
that
point
because
the
gear
was
being
taken
away.
So
that's
where
we
ended
up
with
a
net
bug
is
fixed
in
Coober
Nettie's,
as
well
as
open
ship
33.
At
this
point.
So
if
we
had
the
gear
again,
we
would
get
well
beyond
13,000
projects.
C
B
C
A
C
Yeah
and
sending
you
know,
requests
for
this
type
of
usage
is
nothing
more
than
a
PR
to
the
CNC
f
github
repo.
We
have
some
plans,
as
their
environment
expands
to
start
using
it
again
for
newer
versions
of
our
product.
You
know
so
this
gave
us
a
lot
of
homework
to
do
these
tests
and
these
results
and
it
those
are
not
fixed
overnight.
They
involve
coordination
with
upstream
developers
and
product
ization
on
our
part.
C
So
it's
not
the
type
of
test
we
could
immediately
repeat
anyway,
so
it
kind
of
gives
us
some
think
time
in
the
interim,
while
we're
scheduling
ourselves
for
getting
back
on
the
year
where
you
know
this
was
a
lot
of
stuff
that
we
learned,
I
might
have
a
I'll
just
skip
ahead.
We
filed
somewhere
around
30
plus
bugs
around
this.
This
effort
and
a
lot
of
them
were
in
Cork
urban
areas.
We
found
one
kernel
blog
and
you
know,
but
for
the
most
part,
these
are
solved
at
this
point.
C
I
think
there's
there's
some
outstanding,
but
none
that
are
related
to
scale
and
none
that
we
consider
blockers.
There's
there's
actually
there's
zero
blockers
that
we're
aware
of
on
openshift
33
at
this
point,
given
the
scale
numbers
that
were
quoting
for
support
so
going
beyond
that,
you
know,
there's
dragons
they're
most
likely
and
that's
actually
the
scope
of
our
next
set
of
skill
tests
which
are
working
on
right
now.
It.
C
C
C
Not
just
me
a
more
of
the
coordination
point
of
everything,
so
the
interval
issues
that
we
encountered
we're
between
openshift,
32
and
33.
We
pulled
in
a
new
version
of
Ansul
because
we
wanted
some
of
the
features
and
we
got
those
features,
but
we
also
got
a
performance
regression
and
what
you're
looking
at
is
actually
the
the
after
the
before
the
before
was
so
bad
that
you
could
really
not
even
capture
stats
because
it
never
really
even
completed.
C
The
issue
is
that
the
is
linked
off
there
at
the
top
of
this
slide,
and
you
know
it
was
essentially
a
recursion
of
ansible
fum
tasks
that
was
spiraling
out
of
control
and
just
ending
it
or
just
being
in
these
infinite
loops,
where
every
CPU
is
just
pegged
and
the
system.
Would
this
isn't
what
crash?
Essentially
so
that
bug
is
resolved
in
ansible
22,
which
is
the
version
that
will
be
included
with
open
ship
33.
So
we're
well
beyond
this,
and
you
can
see
now.
C
Actually
the
y
axis
of
this
graph
is
known
as
a
percent
of
course,
so
15
k
is
actually
fifteen
hundred
percent,
which
means
15
course.
This
is
a
16-core
system
for
a
while
there
and
what
it's
doing
at
the
spike
is
actually
all
it's
doing
is
reaching
out
and
collecting
all
of
these
facts
about
a
thousand
nodes,
and
so
that
that
takes
a
lot
of
CPU
to
gather
all
of
that
stuff
together
and
actually
takes
a
lot
of
memory
as
well.
C
So
this
we
couldn't
even
get
to
this
point
in
the
installation
with
some
of
the
other
versions
of
an
school,
so
those
haven't
seen
are
not
supported
on
openshift
there.
They
were
just
find
it
smaller
skills,
but
when
you
start
scaling
out
to
the
thousands,
you
start
seeing
these
issues,
and
so
what
we've
done
is
we've
got
a
version
of
22
that
we're
packaging
with
with
openshift
itself.
A
D
C
D
D
A
B
C
Yes,
it
was
typical,
OSP
stuff
things
we
ran
into
were
because
we
were
using
OSP
ate
everything
we
hit
had
been
already
been
fixed,
and
so
we
pull
some
packages
back.
We
had
a
small
issue
with
heat
OSP
d
needed
a
newer
version,
slightly
newer
version
of
RabbitMQ,
not
0
SPD,
but
the
cluster
need
a
slightly
different
version
of
RabbitMQ,
because
there
were
some
scalability
issues
in
the
versions
that
were
that
we
were
running.
C
C
You
know
we
weren't
necessarily
about
OpenStack
as
much
on
this
test.
We
were
kind
of
just
using
it
as
a
cloud
because
we
wanted
to
get
to
a
thousand,
and
we
only
had
three
hundred
nodes
in
future
versions
of
open
of
this
type
of
testing.
We're
going
to
do
a
lot
more
stack
integration,
for
example,
we've
got.
C
C
C
We're
using
the
products
here
for
a
reason,
the
or
openshift
origin
on
our
do
is
not
something
where
we're
necessarily
focusing
on
we're
more
focusing
on
Cooper
Nettie's
upstream
rather
than
you
know,
orange
and
scalability,
because
we
like
to
fix
the
issues
in
Cuba
as
soon
as
we
possibly
can
and
from
the
OpenStack
standpoint
we
did
the
same
thing
where
we
work.
Everything
upstream
and
Rdio
becomes
an
integration
point.
So
I
don't
I'm
not
aware
of
any
restrictions
around
that
though
so
you
mileage
may
vary,
but
hopefully
you're
lucky.
A
And
if
you
do
it,
then
we'll
definitely
get
you
to
talk
about
it.
Let
us
know
what
happens
there,
anybody
else
out
there
I
know
sundeep
your
microphone
wasn't
quite
working,
so
looked
like
he
popped
off.
I
thought
he
might
pop
back
in
so
I
would
also
moral
pausing
here,
and
people
are
thinking
if
they
have
any
questions.
A
We're
going
to
have
our
first
face-to-face
meeting
of
the
open
shift
and
OpenStack
cig
at
the
open
shift
gathering
november
seventh
at
coop
con
in
seattle,
and
I
typed
into
the
chat
room,
the
link
to
get
to
registering
for
the
gathering,
so
we'd
love
to
have
you
come
Jed
will
be
there
Jeremy's
going
to
be
there
for
part
of
it
and
we're
going
to
have
a
lunch.
We
have
an
hour
and
a
half
set
aside
for
all
the
sig
meetings.
A
A
You
know
more
about
the
load
balancers
or
you
know,
if
there's
a
topic
that's
near
and
during
your
heart
sign
up
to
the
mailing
list,
which
is
on
also
on
the
commons,
that
open
shift,
org
site
and
let
us
know
I-
haven't
created
the
mailing
list
yet
so
that
is
one
of
my
tasks
introduced
today.
So
I
will
add
you
all
to
it
if
you
sign
up,
but
if
there's
anything
else
judge,
is
there
anything
near
and
dear
to
your
heart
that
you
think
we
ought
to
cover
in
the
next
one
yeah.
B
A
couple
things
actually
the
product
ization
of
the
open
shift
on
OpenStack
project
their
codes,
really
good
I
was
wondering
if
you
guys
had
used
that
project
in
your
work
to
create
the
heat
templates
necessary
and
drive
them.
That
was
written
a
question.
How
much
of
the
under
the
/
cloud
heat
templates?
Did
you
have
to
hack
up
from
what
OSP
is
delivering
you
out
of
the
box.
C
So
for
this
test
we
didn't
use
the
heat
templates
for
the
open
shift
on
Italy.
We
didn't
use
the
open
shift
on
OpenStack.
He
templates
that
I
believe
he
was
used
to
deploy
some
of
the
nodes
that
we
had
so
yeah,
but
I
would
say
this
Judd
is
that
we
have
prototyped
openshift
34
on
OpenStack
10
with
those
templates.
We
have
at
least
five
ashes
that
we're
carrying
now
to
make
that
happen.
So
it's
not
not
bad
at
all,
and
our
next
version
of
this
stuff
will
definitely
be
using
those
templates.
B
Cool
because
I'm,
especially
interested
for
my
customers
and
upgrade
paths
and
having
well
defined
templates,
makes
upgrade
somewhat
tractable.
And
you
don't
want
to
have
to
redeploy
your
entire.
Your
entire
network,
in
order
to
upgrade
through
OS,
PS
and
through
open
shifts,
and
even
through,
say,
dell
bios,
updates
or
driver
updates.
B
B
We've
got
a
lot
of
knowledge
here
in
within
dell
about
managing
and
scaling
the
OpenStack.
An
open
shift
networks
on
the
storages
is
is
really
strong
at
Dell,
considering
Lee
we
also
merged
with.
So
we
have
a
wide
variety
of
enterprise
and
SMB
mid-market
storage
products
that
we've
been
testing
with
OpenStack
and
open
shift
to.
B
Let
you
know
how
they're
working,
what
kind
of
integration
is
going
on
see
whenever
I
deploy,
I'm
deploying
on
to
equallogic
boxes
for
all
my
storage
I've
got
a
slightly
different
layout
and
would
like
to
see
where
folks
are
hitting
problems
or
success
stories
around
cinder
integration
and
with
different
storage.
Backends.
B
A
Would
be
great
and
thank
you
everybody
for
for
joining
us
today
and
we
will
I'll
reach
out
to
sundeep
and
see
if
I
can
figure
out
what
this
question
was,
and
we
will
do
another
one
of
these
in
two
months.
So
sometime
in
december
will
pick
a
date
and
we'll
post
it
on
the
calendar.
So
you
can
always
find
the
upcoming
events
on
commons
at
openshift,
org,
slash
briefings,
that's
where
my
calendar
hides
and
we
add
all
the
cigs
and
all
the
briefings
not
coming
events
there.