►
Description
In this week's episode we are taking a look at a project called kube-burner. Kube-burner is a project that can be used for performance and scale testing on Kubernetes/OpenShift Clusters. Andrew & Jonny are joined by Sai, Raul and Vishnu from the OpenShift Performance and Scale team at Red Hat!
kube-burner github: https://github.com/cloud-bulldozer/kube-burner
Join OpenShift Commons:
https://commons.openshift.org/join/
OpenShift Users Slack Channel: https://kubernetes.slack.com/archives...
Want to learn more?
Try Red Hat OpenShift: https://red.ht/3uVxkya
Follow us:
Andrew Sullivan https://twitter.com/practicalandrew
Jonathan Rickard https://twitter.com/jrocktx1
OpenShift https://twitter.com/openshift
Subscribe to Red Hat's YouTube channel: https://www.youtube.com/redhat/?sub_c...
About Ask an OpenShift Admin:
Do you have questions? We want to answer them! Every Wednesday at 11am ET, hosts Andrew Sullivan and Jonathan Rickard sit down to answer your questions about Red Hat OpenShift.
About OpenShift:
Red Hat OpenShift is an open source container application platform based on the Kubernetes container orchestrator for enterprise application development and deployment.
OpenShift on redhat.com https://red.ht/3rYz5ZS
OpenShift on GitHub https://github.com/openshift/
#RedHat #OpenShift #Kubernetes #ContainerPlatform
C
Thank
you,
hello,
and
welcome
to
this
week's
ask
an
openshift
admin
office
hours
live
stream.
I
am
co-host,
Mr
Andrew,
Sullivan
joined
by,
as
always
by
my
lovely
co-host,
Mr
Johnny
Rickard.
How
are
you
today,
sir
I'm
good
Mr
co-host?
How
are
you
today?
It
is
a
stunningly
beautiful
day
outside
I
am
sad
to
say
that
I
am
trapped
in
the
house.
I've
been
doing
most
of
my
work
from
the
back
porch
this
morning.
So
it's
it's
been
pretty
amazing.
That's.
A
C
A
C
A
C
D
C
A
A
C
A
C
C
A
C
Yeah
yeah
well
enough
of
us
rambling,
so
hello,
everyone
welcome
to
this
week's
stream.
So
this
is
one
of
the
office.
Hours
live
streams
here
on
red
hat
live
streaming,
which
means
that
we
are
here
for
you
all
we're
here
to
answer
questions
we're
here
to
talk
about
all
of
the
different
things
relating
to
openshifts,
and
you
know
whether
it's
admin,
stuff
or
Dev
stuff,
although
I
admit
that
Mr
Johnny,
Mr,
Ricard,
Johnny
and
I,
don't
know
a
whole
lot
about
development,
but
we
do
our
best,
but
yeah.
C
Today's
topic
is
something
that
I
I
think
everyone
is
interested
in
is
something
that
comes
up
regularly.
I,
see
constant
questions
in
email
and
chat
and
all
of
the
other
places
I
was
just
looking
at
the
kubernetes
slack.
There's
there's
always
questions
about
performance,
and
so
we
were
super
happy
when
this
team
of
folks
reached
out
to
us
and
said:
hey,
we'd
love
to
come
on.
We'd
love
to
talk
about
coop
burner
and
the
stuff
that
we've
got
going
on.
So
I
am
very
happy
and
I
know.
C
D
Hello,
everybody
I'm,
Sai,
Malini
and
I'm,
an
engineering
manager
so
I
lead
the
performance
and
scale
team
here
at
Red,
Hat
focusing
on
openshift
specifically
and
my
team
does
a
lot
of
work
on
openshift
performance
and
scale.
Obviously,
it's
about
scaling
the
cluster
to
its
limits
and
to
support
our
mission.
We
work
very
closely
with
the
product
management
teams,
the
engineering
teams,
the
QE
teams
and
customer
facing
teams.
I
mean
that
process
build
robust
and
maintainable
open
source.
Tooling,
that's
where
Cube
burner
really
comes
into
the
picture
right.
D
B
A
B
C
E
C
I
I
like
what
you
said:
Psy
there
of
performance
regression
right,
it's
a
huge
thing,
it's
something
that
we
definitely
want
to
avoid
in
many
different
ways:
I
always
look
at
it,
similar
to
what
Johnny's
team
does
with
the
validated
patterns
of
we
want
to
constantly
have
testing
going
on
to
validate
and
verify
that
hey,
we
added
this
new
cool
thing
or
we
we
fixed
this
bug
and
we
didn't
introduce
something
new
or
something
different
or
you
know
unexpected.
Rather
it's
we,
we
can
plan
for
those
things.
C
D
Yeah
this
this
thing's
always
changing
right.
The
proxy
version
changes
going
from
one
openshift
version
to
another
that
may
or
may
not
have
an
impact
on
your
Ingress
performance
or
in
some
cases
the
real
major
version
might
change.
So
we
do
a
lot
of
work
to
catch
these
things
early,
so
operating
as
left
as
possible
right
so.
C
C
A
I'll
just
go
mine
are
gonna,
be
quick,
so
just
again
reminder
for
everybody
that
there's
going
to
be
some
API
deprecations
in
openshift
4.14,
so
just
kind
of
keep
your
eyes
open
for
that.
I
I
did
see
a
Blog
about
the
different
releases
or
about
the
different
features
that
are
coming
out
in
kubernetes
128.
So
I
will
review
that
and
then
next
week
we'll
talk
about
it
and
then
ACS
4.2
was
released.
Also,
there's
a
a
cloud
version.
That's
the
you
know.
A
If
you
wanted
to
get
the,
if
you
wanted
to
kick
the
tires
right,
go
through
a
trial
run
of
it.
You
know
I'll
post
a
link
here
in
just
a
moment
but
yeah.
Basically,
you
can
go
out
and
you
can
try
the
cloud-based
ACS
for
a
little
while
and
then
there
was
an
announcement
earlier
today
or
yesterday.
I
think
about
openstack
services
are
now
going
to
be
available
for
Telco
on
openshift,
which
is
going
to
be
a
pretty
big
deal.
A
That's
that's
kind
of
cool
and
then
finally,
the
what's
new
in
openshift
is
going
to
be
October
12th
at
10
Eastern
to
11
30
Eastern.
So
if
you
want
to
join
Andrew
and
I
are
usually
there
in
the
background,
answering
questions
so
I
again:
October
12th,
10
to
11,
30,
Eastern
and
then
finally,
for
me,
the
upcoming
schedule
is
next
week
we
got
the
the
man
in
the
myth,
the
legend
Orton
coming
on,
to
talk
about
all
things:
surface
mesh
and
then
in
October.
A
C
Yep
I,
honestly
haven't
looked
to
see
because
I'm
not
I'm,
not
going
I
wish
I
was
going.
I
haven't
been
to
a
kubecon
in
a
couple
years,
but
yeah
we'll
see
So
yeah.
Thank
you.
Let's
see
what
did
I
have
on
here.
I
had
a
couple
of
things,
so
two
things
came
up
in
conversations
recently,
so
the
first
one
I'm
gonna,
share
my
screen
here,
because
that's
gonna
be
the
easiest
way
to
talk
about
it.
C
Share
this
guy.
So
the
first
thing
that
I
want
to
bring
up
here
is
the
high
availability
flag
that
sets
when
deploying
a
cluster.
So
there's
kind
of
two
deployment
models
for
openshift
right.
There
is
the
traditional
three
or
more
node
cluster
deployments,
so
I
have
a
compact
cluster.
So
it's
three
nodes
where
all
the
nodes
are
both
control,
plane
and
worker
nodes,
and
then
I
have
five
node
clusters.
C
So
three
control
plane
nodes,
two
worker
nodes
and
then
the
other
side
is
either
single,
node,
openshift
or
a
cluster
where
there
is
only
one
worker
node.
So,
for
example,
it's
possible
to
deploy
a
cluster
that
has
three
control
plane
nodes
and
a
single
worker.
Node,
when
you
do
either
one
of
those
ladder,
deployment
models
or
cluster
architectures,
what
it
does
is
it
changes
this
flag
right,
this
infrastructure
topology
to
be
single
replica.
C
So
what
does
that
actually
mean
so
that
flag
is
used
by
various
operators
that
make
up
openshift
in
order
to
identify
and
determine
kind
of?
What
is
the
availability
constraints
that
I
have
available
to
me
so,
for
example,
with
the
Ingress
controller
right,
so
the
AJ
proxy
based
Ingress
for
bringing
traffic
into
openshift?
C
C
C
Generally
speaking,
this
isn't
an
issue
most
of
those
operators
that
you
can
go
in.
You
can
change
them.
So,
like
I
said
the
Ingress
operator,
you
can
go
in
day
two
after
you
add
those
worker
nodes
and
scale
up
the
number
of
replicas
to
provide
High
availability.
Probably
the
most
important
one
is
openshift
virtualization.
C
C
So
just
keep
that
in
mind
if
you're
doing
openshift
virtualization,
if
you're
using
single
node
openshift,
you
know
planning
on
adding
worker
nodes
afterwards.
I'll
also
point
out
that
this
doesn't
change.
You
know
quote
unquote
to
the
high
availability
of
workloads
inside
the
cluster.
So
if
you
deploy
single
node
openshift
and
then
you
add
some
worker
nodes,
if
one
of
those
worker
nodes
were
to
fail,
it's
going
to
reschedule
those
failed
pods.
Just
like
always
right.
C
It
doesn't
change
any
of
the
behavior
in
that
respect,
yeah,
so
the
other
thing
that
I
will
quickly
point
out
is
this
came
from
a
conversation
and
actually
cyan
company.
You
might
be
a
good
crowd
to
add
in
some
some
comments
if,
if
they
come
to
mind,
so
somebody
was
asking
about
actually
the
number
of
routes
that
are
supported
by
the
AJ
proxy
based
Ingress,
right,
I,
think
by
default,
and
if
we
come
down
here,
come
on
what
else?
If
we
scroll
down
through
here
to
scalability
and.
C
It
should
yep
yep,
so
if
we
look
down
here
right,
2
000
per
per
shard,
so
the
question
was:
why
hasn't
this
changed
in?
However
many
versions,
you
know
why?
Why
is
it
not
supported?
To
put
you
know,
2001
on
there
or
something
like
that
and
the
response
to
that
is
interestingly.
Well,
it's
not
unsupported.
C
C
The
difference
is.
We
know
that
up
to
2000
per
router
is
going
to
work.
We
test
that
as
I
said,
so
we
generally
know
exactly
what
to
expect
what
the
limitations
are
when
there
might
be
issues
right,
barring
edge
cases
and
stuff
like
that,
and
so
that's
also
what
they
engineer
against.
So,
if
you
put
10
000
inside
of
there,
it
might
work
perfectly
fine.
C
There
might
not
be
any
issues
and
it'll
do
exactly
what
you
expect
it
to
do,
and
you'll
never
have
to
pick
up
the
phone
and
call
us,
but
it
might
not,
you
might
encounter
some.
You
know
Edge
case
that
we've
never
encountered
before
and
supports
response
might
be
well
I'm.
Sorry,
you
know
your
best
option
right
now
is
to
reduce
it
to
2000
per
router
or
fewer.
C
Conversely,
as
I
said,
there
are
sometimes
edge
cases.
Maybe
you
only
have
500
routes
on
that
router,
but
they're
constantly
changing
and
churning,
which
is
causing
H.A
proxy
to
constantly
have
to
reload,
and
it
results
in.
You
know
that
that
router
not
being
able
to
support
up
to
two
thousand
so
the
little
secret
behind
the
scenes
there
is
so
the
the
proxy
config
is
reloaded
every
time
one
of
those
routes
changes.
C
So
what
causes
a
route
change
right,
clearly
doing
something
like
creating
a
new
routes
or
just
deleting
a
route
modifying
Services,
the
pods.
So
let's
say
I've
got
a
route
that
has
a
thousand
pods
that
it's
pointing
to.
If
those
pods
are
constantly
bouncing
around
up
and
down
it's
having
to
change.
It's
having
to
update
that
route
config
to
know
where
to
direct
that
traffic
to
so
that
can
cause
that
number
of
routes
per
router
to
be
lower
than
you
might
expect.
C
D
C
I
I
I
will
very
happily
admit
that
I
used
to
do
a
lot
of
performance
testing
when
I
was
a
storage
admin,
as
you
might
expect,
but
anyways,
that's
all
I
got
Johnny
covered
the
vast
majority
of
things
so
I
just
wanted
to
hit
on
those
two
quick
points,
because
I
think
it's
important,
particularly
for
this
tested,
maximums
right
tested,
does
not
mean
the
same
thing
as
the
floor
or
ceiling
as
the
case
may
be
for
supported
all
right
with
that.
We
have
consumed
more
than
enough
time.
C
So
please
feel
free
to
ask
any
questions
that
are
that
come
to
mind.
You
know,
ask
those
in
the
chat
anything
that
you
would
like,
whether
it's
about
performance
and
scale,
whether
it's
about
openshift
we'll
do
our
best
to
answer
all
of
those
questions.
If
you
aren't
watching
us
live
I
know
a
lot
of
people
like
to
listen
to
the
recordings.
You
can
also
feel
free
to
reach
out
to
us
afterwards,
and
we
will
be
more
than
happy
to
answer
those
questions
to
the
best
of
our
ability.
C
So,
for
reference
you
can
reach
out
to
I
can
never
find
the
button
that
I'm
looking
for
when
I
want
to
find
it.
You
can
reach
out
to
me
at
andrew.sullivan
redhat.com
or
on
the
various
social
platforms
at
practicalandrew,
or
you
can
reach
out
to
Johnny
via
email
at
johnnyjonny,
no
h
at
redhead.com
or
Jrock
tx1
across
all
the
various
social
platforms,
and
with
that
I'm
going
to
hand
it
over
to
you
Psy,
so
performance
and
scale.
D
Awesome
so
again,
we've
just
introduced
ourselves,
so
I
don't
feel
the
need
to
do
it
again,
but
we're
going
to
talk
about
Cube
burner
and
what
is
Cube
burner
right.
The
one
line
summary
is
it's
a
kubernetes
performance
and
scale
test
orchestration
framework
right.
We
we
use
it
against
openshift,
but
it's
more
than
that.
We
can
run
it
against
Manila
kubernetes
against
different
districts
of
kubernetes,
but
the
work
that
my
team
does
primarily
is
focus
on
openshift
and
there's
a
few
pieces
to
its
functionality
right.
D
One
is
its
ability
to
create,
read,
update
and
delete
kubernetes
resources
at
scale.
So
it's
it
helps
us
test
the
cluster
to
its
limits.
It
is
also
able
to
go
and
collect
Prometheus
metrics.
While
we
run
these
tests
and
index
them
into
something
that's
more
of
a
longer
term
storage
like
elasticsearch
or
open
search.
C
D
C
It
does
and
I
want
to
point
out
that
it's
very
good
that
there
is
canned
workload
available,
but
you
know
a
lot
of
times.
You
know
I
respond
to
a
lot
of
customers
of
like
oh
you
know.
What
does
it
mean
when
I
do
this
there's
actually
an
open
thread
somewhere
inside
of
red
hat
about
you
know
the
difference
between
openshifts
Java
based
applications
hosted
an
open
shift
on
bare
metal
versus
virtualization,
and
it's
one
of
those
like
there's
so
many
factors
there.
D
We
have
we
have
various
success
stories
where
we've
used
this
tool
to
give
customers
data,
but
not
just
that
right,
empowered
customers
to
model
their
own
workloads
and
see
how
they
fail
so
yeah
all
right
with
that.
I
will
just.
Why
do
we
need
Cube
winner
right?
Some
of
the
questions
that
we're
trying
to
answer
with
couponer?
D
D
You
must
have
noticed
in
the
openshift
documentation
that
it
used
to
say
250
back
in
the
day
and
it's
still
250
out
of
the
box,
but
we've
done
several
experiments
using
Cube
burner
to
prove
that
you
can
go
all
the
way
up
to
2500
parts
per
note
again,
your
mileage
might
vary.
There
are
a
lot
of
conditions
you
have
to
satisfy.
You
have
to
keep
monitoring
the
load
on
the
system,
but
like
everything
with
performance,
it's
subjective,
but
kubernet
has
helped
this
model
that
workload
and
push
the
runtime
to
the
limits.
D
How
many
crds
can
my
cluster
handle?
This
is
related
to
one
of
the
questions
we
see
on
chat.
How
many
operators
can
my
cluster
handle,
which
is
related
to
how
many
crds
you
can
throw
on
a
control
plane
there
right?
How
long
does
the
cni
plug-in
OV
and
kubernetes
are
openshift
sdn?
How
long
does
it
really
take
to
program
those
floors?
How
many
services
can
I
create
is
5000
too
many.
Does
it
matter?
D
If
it's
a
note
port
or
you
know,
cluster
IP
type
of
service
is
kubernetes
getting
better
with
each
release
or
is
it
getting
worse?
Has
there
been
any
performance
degradation
when
we
bump
the
hcd
version?
So
all
of
these
questions
they
come
to
mind
and
Cube
bundle
is
one.
You
know
Swiss
army
knife
that
can
help
you
answer
some
of
these
questions
and
it's
not
just
for
performance,
Engineers,
I.
D
Think
if
you
look
at
the
different
personas,
if
you're
an
architect
and
if
you
want
to
figure
out
what
the
cluster
sizing
should
be
like
and
if
you
want
to
do
some
capacity
planning
or
establish
some
slos
that
are
actually
tangible
and
real,
you
can
use
kubernet
to
do
that
as
an
admin.
You
want
to
know
what
the
Baseline
performance
of
your
cluster
is
right.
What
are
the
slis
looking
like
and
Cube
burner
can
help
you
with
some
of
that
as
an
app
developer.
A
It's,
not
just
engineering
right,
it's
actually
for
these
different
groups
because
they
they
have
to
know
and
but
then
my
my
question
is
I
I
know
you're
doing
a
lot
of
this
with
openshifts
and
stuff
like
that.
But
are
you
contributing
this
back
to
like
Upstream
kubernetes,
like
is
the
there's
kubernetes
project
or
the
umbrella,
the
cncf
umbrella,
using
this
at
all.
D
So
there
is
a
proposal
in
progress,
so
we
have
a
CNC
at
sandbox
application
already
in
Flight.
But
yes,
our
primary
testing
is
an
openshift.
We
actually
have
kubernet
running
in
Cloud
jobs,
but
I
am
pretty
sure.
People
are
using
this
to
run
it
against
vanilla
communities
and
we,
for
example,
our
CI
is
based
on
kind
right,
so
we
run
it
against
vanilla
kubernetes.
D
E
All
right,
thanks,
eh
now
at
this
moment
you
might
be
having
questions
on.
How
does
kubernet
work
like
how
it
actually
manages
to
connect
to
the
cluster,
how
it
actually
creates
resources
and
triggers
a
workload
and
how
it
you
know:
scrapes
the
metrics
and
publishes
them
or
indexes
them
to
a
desired
destination.
So,
in
order
to
answer
these
questions,
slow
here
is
the
slide.
E
So
we
we
ship
all
our
source
code
in
the
form
of
a
gobinary
like
executable
binary,
which
is
available
for
all
architectures,
and
these
are
some
of
the
libraries
which
we
extensively
use
in
our
source
code.
One
is
client
group,
so
using
this
Library,
we
managed
to
connect
to
the
cluster
using
group
config,
and
also
we
make
API
endpoint
hits
to
API
server
endpoints.
E
So
these
hits
help
us
to
you
know,
trigger
the
resources
and
manage
them
throughout
the
life
cycle
and
throughout
the
workload,
and
we
also
make
use
of
Prometheus
go
client
which
helps
us
to
connect
to
any
Prometheus.
Not
only
it
need
not
necessarily
be
to
the
cluster
Prometheus
in
order
to
scrape
the
metrics
and
also
we
run
a
set
of
queries
to
make
sure
like
we
capture
all
the
metrics
during
the
workload
runtime,
and
also
we
make
a
record
of
them.
B
E
B
E
And
also
Cube
burner
currently
supports
all
kinds
of
gates
distributions,
because
it
uses
some
of
the
base
core
libraries
in
order
to
do
its
operations,
it
can
support
kids,
any
case
distribution,
like
kind
or
in
any
kubernetes
cluster,
also
a
whole
bunch
of
cluster
as
well,
but
most
of
our
activity
is
involved
in
an
openshift
cluster.
So
few
few
more
points
about
couponer
are
like.
It
is
very
easy
to
play
around
with
you.
You
just
need
to
specify
a
yml
file
as
an
input.
E
Configuration
and
coupon
is
the
one
which
consumes
it
and
then
triggers
the
workload
based
upon
the
configuration
parameter
specified
and
also
once
we
have
the
workload
triggered
and
run.
We
capture
the
measure,
and
then
we
also
capture
some
of
the
cluster
Prometheus
metrics.
In
order
to
look
at
the
cluster
from
performance
during
the
workload
and
once
we
have
all
those
metrics
ready,
we
index
them
to
our
indexes,
which
can
either
be
a
local
file
system,
or
else
a
remote
indexer
like
openshift
or
elasticsearch.
E
So
here
is
the
overview
on
the
kubernet
workflow.
So
here
is
where
it
starts.
So,
for
example,
if
you
look
at
the
first
step,
is
Cube
burner
would
be
the
entry
point
and
we
will
be
specifying
a
yml
file
as
an
input
configuration.
So
it
reads:
the
yml
file
keeps
it
in
memory
and
based
upon
the
configurations
which
we
specify
over
there.
It
creates
the
resources
First
Once.
E
The
resources
are
created,
kubernet
manages
to
identify
the
resources
which
are
created
as
a
part
of
workload,
ignoring
the
rest
of
them
in
the
cluster
and
then
triggers
measurements.
On
top
of
it,
it
kind
of
keeps
a
watcher
on
top
of
those
resources
to
as
a
starting
point
to
capture
the
measurements.
E
So
once
that
is
done,
it
triggers
the
workload
and
waits
for
the
expected
state
to
be
reached.
For
example,
if
you
have
a
set
of
quartz
being
spinned
up,
it
waits
for
ports
to
get
ready
and
also
we
have
a
custom
feature
of
specifying
our
own
condition
for
termination.
For
example,
if
you
want
to
specify
some
like
wait
until
pots
reach
container
steady
state,
it
can
wait
till
there
and
then
you
know,
stop
measurements
and
then
take
it
forward.
E
So
once
it
reaches
the
expected
state
of
the
workload,
then
we
stop
our
measurements
on
all
the
resources
calculate
perform
some
statistic:
calculations
on
top
of
them,
you
know
together,
latencies,
P,
Prof,
Matrix
or
any
other
Matrix,
which
we
specify,
and
once
you
have
this
metrics,
we
also
make
sure
we
scrape
custom
with
this
Matrix
as
well.
So,
during
this
workload
trigger
period,
we
make
a
note
of
start
and
end
time
and
during
those
time
periods
we
run
a
bunch
of
queries
on
cluster
Prometheus,
which
basically
help
us
indicate
and
evaluate
cluster
format.
E
Cluster
performance
during
the
workload
was
ran
so
once
we
have
all
these
runtime
metrics
as
well
as
cluster
Matrix
ready,
we
manage
them
to
index
to
our
indexes
So.
Currently
we
have
three
types
of
indexes:
supported,
open
search,
elasticsearch
as
well
as
local
indexer.
Local
indexer
is
nothing
but
we
index
those
metrics
into
our
local
file
system
in
Json
format
and
also
along
with
the
alerting
feature
along
with
the
scraping
feature.
We
also
have
some
alerting
feature
that
can
be
configured
wherein
we
specifies
a
set
of
configuration
to
trigger
I
mean
based
upon
thresholds.
E
We
trigger
alerts,
and
then
we
also
we
can
also
make
a
note
of
those
alerts.
So,
for
example,
we're
just
looking
at
the
metrics
we
know
might
not
be
able
to.
You
know
grasp
all
that
information
at
once,
but
having
these
alerts
configured
it
will
help
us
to.
You
know
immediately
look
at
what
we
actually
were
expecting
to
look
at.
So,
for
example,
if
we
have
set
like
you
know,
pod
latency
threshold
of
you
know
these
many
seconds.
E
E
So
in
create
job
it
reaches
the
configuration
file
and
creates
all
the
resources
triggers
workload
on
top
of
them,
and
then
you
know,
collect
symmetics
and
publishes
through
the
desired
indexer
like
the
workflow
indicates
below
when
it
comes
to
delete
job.
It
identifies
all
the
resources
based
upon
the
labels.
We
have
specified
label
selectors
and
then
deletes
them
and
when
it
comes
to
pad
job,
does
follow
the
same
pattern
of
identifying
all
the
resources
based
upon
label
selectors
and
applies
the
patches
which
we
specify
through
the
configuration
file.
E
B
Thanks
Vishnu,
so
regarding
key
partner
perform
measurements.
That
is
something
that
is
not
already
mentioned.
The
difference
between
your
brother
measurements,
on
the
kind
of
the
standard,
Prometheus
Matrix,
is
that
these
measurements
do
not
have
any
need
of
having
Prometheus
in
stands
running
on
top
of
your
cluster.
These
kind
of
measurements
are
directly
taken
from
data
points
obtained
from
the
API
or
other
subsystems.
At
the
moment,
we
are
supporting
three
kind:
three
different
type
of
measurements,
which
are
basically
both
latency
VMI
latency
and
people
of
metrics.
B
A
B
Installed
in
a
local
directory
in
people
format
and
use
that
data
to
plot
a
nice
flame,
graphs
or
analyze
or
deep
analyze,
if
there's,
if
there
has
been
any
kind
of
memory
leak
or
something
like
that
in
any
kind
of
fabrication,
running
any
kind
of
golden
application
running
in
in
openshift.
This
is
this
is
mainly
meant
to
to
be
triggered
against
control,
play
components
of
openshift
or
kubernetes.
Like
a.
C
I
would
imagine
all
of
these
measurements
are
used
in
conjunction
with
what,
for
example,
etcd's
own
performance
reporting
is
doing
so
as
we
ramp
up
and
add
more
objects
or
more
activity
happening
in
the
cluster.
We
also
compare
that
against
what
was
happening
in
NCD
at
the
same
time,
so
we
can
get
that
idea
of
yeah.
B
Imagine
imagine
this
case.
Imagine
you
you.
We
have
detected
that
that
City
memory
usage
has
grown
a
lot
in
a
new
version
of
openshift
and
we
have
been
asked
by
the
LTD
team
to
figure
out
where
that
memory
usage
is
coming
from.
So
one
way,
apart
from
counting
the
number
of
keys
and
investigating
some
of
the
database,
content
in
entity
would
be
to
get
a
Prof
during
a
workload.
So
we
can
see
what
internal
goal
and
structures
have
grown
during
this
new
version.
C
B
Exactly
exactly
in
in
our
grouping,
we
used
to
say
that
every
object
has
a
performance
impact
regardless
that
object
is
a
complex
object
like
a
CRT
or
something
like
a
CD
config
Mark
doing
anything,
because
every
object
is
stored
in
entity
and
could
be
potentially
listed
by
the
API
or,
and
that
is
going
to
consume
some
CPU
cycles
and
also
some
block.
I
o
Cycles
as
well.
B
Supported
ones
are
elastic
certs
and
we
are
also
using
open
search,
which
is
also
offered
by
AWS,
and
we
are
also
supporting
Json
files,
which
can
be
useful
at
the
time
of
executing
your
partner
in
disconnected
environments
in
case,
for
example,
your
clients,
the
client
machine,
where
they
are
used,
where
you're
executing
couponer
front
as
an
access,
doesn't
have
access
to
any
kind
of
elastic,
certs
or
open
searching
instance.
You
can
basically
set
those
files
into
Json
and
import
them
later
for
not
connected
environment
and
yeah.
B
That
is
to
say
that
if,
for
example,
your
cluster
have
a
couple
of
rubies
of
monitoring
stocks,
let's
say
you
have
the
you
say,
monitor
in
the
stack
set
up
in
your
environment.
Your
partner
is
also
able
to
scrape
from
the
main
promises
instance,
and
also
from
the
user
monitoring
stack
instance
and
yeah.
Coupon
is
also
able
to
generate
a
terrible
containing
ultimatics
so
that
we
can
import
them
later
now
in
a
connected
environment.
D
Yeah
I
think
that
kind
of
brings
us
to
the
brief
overview
we
want
to
do
in
terms
of
a
slide
deck,
which
should
give
us
some
time
to
get
into
the
fun
part,
which
is
the
demos,
if
that's
okay,
with
you,
Andrew
and
Johnny,
or
if
you
want
to
include
a
few
questions
before
that.
That's
totally
fine,
too.
That.
B
Go
ahead
last
week,
sorry
I
forgot
to
talk
about
the
alerting
feature,
so
alerting
feature
is
something
kind
of
similar
to
Prometheus
alertin.
It
is
basically
the
feature
that
makes
kubernet
able
to
reach
something.
We
call
metrics
for
alert
profile
and
into
automatic
profile
is
just
that
now
right.
A
list
of
promises,
expressions
and
qpernel
will
take
care
of
reading
those
promises,
Expressions
executing
them
into
the
selected
Prometheus
endpoint,
and
get
the
result
of
the
of
those
expressions.
B
D
B
D
Yeah
API,
availability
and
stuff
during
the
workload
we
could
just
get
an
alert
that
says:
hey
your
API
was
not
available,
for
you
know
x,
amount
of
curation,
so
we
have
some
demos
plan.
Like
I
said
we
may
not
have
time
to
get
through
everything,
so
we'll
probably
get
through
a
few
of
them.
We
had
five
demos
planned.
We
probably
will
hope
to
get
through
at
least
three
of
them
awesome
all
right.
So
hopefully
my
screen
sharing
works.
D
D
Okay,
so
I'll
start
with
the
bare
bones:
functionality
and
we'll
build
up
through
a
series
of
demos
to
the
full
functionality,
and
what
I'm
going
to
show
is
the
basic
sub
Command
right,
the
init
sub
command.
It
is
the
entry
point
for
kubernet
to
launch
benchmarks,
and
the
only
argument
it
really
expects
is
a
config
file
which
really
gives
you
a
lot
of
control
over
what
you
want
defined
to
be
run
by
kubernet.
The
uuid
argument
is
optional.
D
If
you
don't
pass
something
it
will
just
Auto
generate
it
and
the
reason
we
need
this
is
kuberneter
is
going
to
label
all
the
resources
it
creates
with
this
uuid.
So
it's
easy
for
you
to
identify
what
coupon
are
created
versus
what
you
know
something
else
created.
So
this
is
the
basic
functionality.
It
creates
resources,
it's
able
to
delete
resources.
D
We
look
at
what
the
config
file
looks
like
and
before
that,
what
am
I
trying
to
really
do
here,
I'm
trying
to
create
10m
spaces,
each
of
them
having
three
deployments
and
each
deployment
having
one
replica
of
an
nginx
part.
So
that's
really
what
I'm
trying
to
do
here
and
the
kubernet
config
file
is
divided
into
a
series
of
jobs.
You
can
have
multiple
jobs
in
a
list.
D
I
only
have
one
job
here
right,
that's
what
the
jobs
keyword
here
denotes,
and
there
are
some
Global
options
here
like
GC
through
which
means
hey
Cuban
or
you
should
clean
up
after
yourself.
Go
garbage,
collect
everything
you've
created,
so
that's
something
you
can
set
to
false
or
true
in
the
interest
of
time.
I
will
actually
set
it
to
true
so
that
it
can
clean
up
after
itself
or
we'll
see.
That's.
D
Yeah,
thank
you
so
yeah.
So
let's
just
look
at
the
job
configuration
right.
It's
just
the
name
of
the
job.
The
ask
ocp
admin.
You
know
have
that
special
name
for
you
guys.
What
is
this
prefix
of
the
name
spaces
you
want
to
create
Ask
ocp
admin
will
be
the
prefix
I
want.
10
iterations,
which
means
create
tenant
spaces.
Qps
and
burst.
Are
client
go
parameters
that
determine
the
rate
at
which
Q100
operates
against
the
kubernetes
API.
D
So
it's
basically
creating
five
objects
per
second
namespace
iterations
means
for
every
job,
iteration
create
a
new
namespace,
so
your
namespace
is
going
to
look
like
ask.
Ocp
admin
zero,
one,
two
three,
so
it's
going
to
spread
things
out.
Wait
when
finished
through
means,
wait
for
all
the
pots
to
be
running,
so
you
can
finish
the
measurement
of
how
long
it
took
for
a
pot
to
be
ready.
D
Preload
images,
true
means
the
first
container
running
on
a
particular
node,
is
always
going
to
take
more
time
because
you
need
to
do
the
image
download
right
and
that
will
skew
up
our
measurements.
So
what
this
tells
kubernator
is
just
create
a
demon
set
with
this
image.
Is
that
all
the
images
you're
going
to
use
in
this
run
and
make
sure
they're
running
before
you
run
the
actual
workload?
A
lot
of
this
is
boilerplate.
You
don't
necessarily
have
to
Define
all
of
these.
D
When
you
run
this
config
file,
we
will
assume
save
defaults
as
needed.
The
real
thing
that's
happening
here
is
creating
three
objects.
Well,
the
objects
list.
We
are
only
creating
deployment
three
deployments
per
namespace
across
10
namespaces
right
and
the
replica's
argument
here
is
specific
to
cube
burner.
It
does
not
denote
the
replicas
of
the
deployment,
it's
how
many
replicas
of
the
object
kubernet
is
creating.
You
want
right,
and
it's
going
to
use
this
template
to
create
it.
D
Let's
take
a
look
at
that
template
real,
quick
to
see
what
it's
it's
really
just
an
nginx
deployment
with
replica
one.
Some
of
these
variables,
like
replica,
are
injected
by
Cube
burner
to
uniquely
distinguish
the
different
deployments.
It's
going
to
create.
Remember
it
creates
three.
The
problems
for
namespace
every
deployment
needs
to
have
a
unique
name
right.
So
it's
going
to
just
inject
this
automated
variable,
which
is
the
replica
and
you
can
use
that
and
all
of
this
happens
through
ghost
templating
language.
So
we
do
that.
D
I
have
my
nice
little
Rosa
cluster
right
sitting
in
the
cloud
for
three
infinite:
three
Master
Three
working
nodes:
let's
see
what
Cube
burner
in
it
is
gonna.
Do
when
I
run
this
the
starting
Cube
on
Earth.
It's
doing
the
preload
of
the
images
right.
I
want
to
do
some
measurements
along
how
long
it
took
for
the
part
to
go
through
scheduling
all
the
way
to
Containers
ready.
So
for
that
I
don't
want
to
skew
the
measurements.
D
So
the
first
one
is
being
created
as
a
demon
set
across
all
the
worker
nodes
available
on
the
cluster.
Then
it's
deleting
that
and
it's
running
those
job,
iterations
right,
10
namespaces!
It's
going
to
go
and
create
it.
I
asked
for
it
to
check
that
all
the
pods
were
created.
So
it's
going
through
and
doing
that
as
we
speak
and
I
think
I
said:
GC
equals
to
true
yeah,
I
I.
Think
I
left
GC
equals
to
true.
D
So
it's
cleaning
up
after
itself,
but
if
I
had
GC
set
to
false
I,
would
have
to
use
a
different
command
which
is
called
kubernet,
destroy
to
go
through
and
destroy
all
the
namespaces
that
it
created.
Let's
read
further
still
like
finish
here
in
a
second,
but
any
questions.
While
we
see
this,
and
while
we
have
it,
wait
for
it
to
complete.
C
I
I
just
think
that
this
is
fascinating
for
a
number
of
different
reasons,
one,
it's
very
straightforward
right.
It's
yamo!
It's
the
same
sort
of
templating
language
that
we're
used
to
across
you
know
multiple
other
Red,
Hats,
I'll,
say
products.
You
know,
and
projects
and
stuff
like
that.
So
it's
very
straightforward
to
to
look
and
to
see
exactly
what's
going
to
happen
inside
of
there
yeah.
D
C
D
This
is
the
bad
bonds.
Functionality,
Raul
and
Vishnu
will
build
into
this
functionality
because
we
scrape
all
of
these
measurements
and
metrics
and
ship
it
to
elasticsearch,
and
we
could
just
have
a
grafana
front
end
hooked
into
that
backend
to
graph
all
of
this
stuff.
So
I'm
gonna
pass
it
off
to
tuition
in
the
interest
of
time,
but
yeah
I
had
GC
sent
it
through.
I
did
I
forgot
to
say
false,
but
if
I
had
GC
set
to
false
I
would
have
you
know
Cuban
and
Destroy
pass
it.
D
D
B
A
D
E
C
E
So
I'll
be
demonstrating
a
different
option
here,
so
I'll
be
demonstrating
kubernet
along
with
indexing.
So
this
is
how
the
configuration
file
looks
like,
which
is
similar
to
the
previous
broilerplate,
which
we
took
a
look
at
and
just
zoom
in
a
little
bit.
Okay,
so
here
is
where
we
split
for
our
Global
parameters.
E
We
say:
DC
is
equal
to
true
to
garbage
color
to
you
know,
to
enable
The,
Graduate
connection
step
as
well
as
we
specify
the
indexer
configuration.
So
here
we
say
it
as
local.
That
means
to
index
all
the
metrics
to
our
local
file
system,
and
also
here
we
talk
about
measurements
that
needs
to
be
calculated
during
the
workload.
So
here
we
specify
pod
latency
along
with
the
threshold,
so,
for
example,
if
it
breaches
over
a
threshold
of
P95
P99
greater
than
10
seconds,
it
will
drop
us
an
alert
and
and
along
with
that
it.
E
This
is
the
metric
which,
which
is
need
to
be
calculated
similar
to
pod
latency.
We
can
specify
VM
latency
and
P
Prof,
as
we
mentioned
in
the
previous
slides
and
let's
move
on
to
the
jobs
part.
So
this
here
is
where
the
magic
happens
here
is
where
we
configure,
which
you
know
the
name
of
the
job.
You
know
the
type
of
the
job
by
default.
It
is
create
type.
This
job
creates
all
the
resources
and
triggers
workload,
and
here
is
the
namespace
prefix,
and
here
is
100
iterations
of
these
objects.
E
So
this
will
be
a
generic
template
and
objects
is
something
which
user
is
empowered
to
play
around
with.
It
need
not
necessarily
be
necessarily
be
a
deployment
and
secret.
It
can
either
be
a
crd
and
any
you
know,
API
resource
which
is
supported
by
kubernetes,
so
it
just
does
the
operation
which
is
similar
to
OCR
play
to
just
apply
those
or
cube
CTL
apply
to
apply
those
resources
on
top
of
the
cluster.
E
So
that's
one
more
interesting
aspect
of
it
of
this
product,
and
this
is
QPS
which
basically,
you
know,
indicates
the
queries
per
second.
The
couponer
can
make
to
the
API
server
to
hit
the
endpoints,
and
you
know
manage
those
resources
either
create
patch
you
know,
perform
all
the
credit
operations
and
burst
is
an
interesting
feature.
What
it
indicates
is
whenever
there's
a
spike
situation
in
to
the
to
the
API
server,
on
the
request
being
made
on
how
many
Mac,
on
how
many
maximum
amount
of
request
to
throttle
on.
E
For
example,
we
are
hitting
API
server
with
10
QPS
and
if
there's
a
spike
situation
on
the
API
server,
it
can
throttle
up
to
10
more
requests
and
the
remaining
ones
it
will
try
to
manage.
But
you
know
if
they
fill
up
the
work
queue,
they
will
eventually
get
time
dot
or
failed.
So
that's
the
significance
of
this
burst
aspect,
but
most
likely,
if
we
set
both
of
them
too,
it
should
be
a
smooth
and
peaceful,
run
and
name.
Space.
C
A
E
So
with
all
this
configuration,
let's
go
ahead
and
in
order
to
get
the
workload
and
one
more
good
thing
about
this,
along
with
this
configuration,
we
also
managed
to
specify
something
called
a
matrix
profile
which,
which
is
nothing
but
has
a
set
of
Prometheus
queries
which
which
we
were
talking
about
capturing
cluster
Prometheus
metrics
after
the
workload
you
know
to
check
the
performance-
and
you
know
behavior
of
the
cluster
during
the
workload
period
And.
This
is
the
flexibility
that
a
user
can
have.
They
can
specify
any
query.
E
A
So
while
you're
running
that
really
quick
I'm
just-
and
this
is
just
more
of
a
general
question
like
I
know,
it's
it's
a
project,
and
this
is
a
binary
and
stuff
like
that.
Right.
Are
you
building
this
into
an
operator
and
going
to
deploy
it
as
an
operator,
or
is
it
always
going
to
be
something
that
you
would
just
pull
down
and
run
kind
of
externally
to
the
cluster.
E
So
we
usually
yeah
go
ahead.
Sorry,
so
we
usually
generate
a
binary
and
using
that
bandwid
trigger
our
workloads.
So
when
it
comes
to
scale
aspect,
you
know
triggering
it
through
a
CI.
You
know
we
have
some
scripts
boilers
like
the
template.
Scripts,
which
manage
to
you
know,
generate
a
configuration
file,
template
which
is
designed
by
couponer
and
triggers
the
you
know
workload
through
passing
it
as
an
input
to
the
binary.
D
A
So
that's
that's
kind
of
like
where
my
question
was
coming
from
so
then
my
next
question
is
really
more
of
like
future
requesting
I
guess,
maybe
or
so,
are
you
guys
going
to
generate
like
templated
I
know,
you've
got
these
templates
for,
like
your
deployments
and
stuff
like
that,
but
like
a
GitHub
actions,
template
or
like
a
tecton
task
template
or
something
like
that
that
would
have
like
this
cool
burner
stuff
in
there.
So
that
way,
all
you're
doing
is
you're
just
dumping
in
params
and
then
letting
it
go.
A
A
You're
at
no,
no,
like
so
say
if
I
have
openshift
pipelines
installed
and
I
want
to
run
this,
like
hey
I,
want
to
do
a
performance
test
of
an
application
that
I'm
doing
is
there
like?
Are
you
guys
going
to
generate
like
a
task?
That's
going
to
be
public,
that
I
can
pull
down
and
then
I
can
just
say:
hey
here's,
the
namespaces,
here's
the
number,
here's
the
diversity
and
stuff
like
that
and
then
go.
A
D
E
So
we
just
have
to
extract
those
templates
and
you
know
make
tweaks
on
top
of
it
if
there
is
any
custom
configuration
which
someone
designs,
yeah
yeah.
So
let's
look
at
this
run
in
detail,
so
here
I
just
specified
couponer
unit
command
with
the
configuration
file
which
I
talked
about
earlier
and
the
Matrix
profile
file,
which
has
a
bunch
of
Prometheus
queries
to
scrape
at
the
end
of
workload
and
Prometheus,
URL
and
token,
as
well
as
the
log
level.
E
So
here
since
we
have
enabled
debug
logging,
which
is
one
more
interesting
feature
which
you
can
you
know,
look
in
detail
like
what
happened
during
the
workload,
so
it
just
created
a
indexer
object
to
index
all
our
metrics.
That
is
that's
just
you
know,
creating
a
directory
in
the
local
file
system
and
then
you
know,
establish
a
connection
with
Prometheus
and
then
start
the
workload.
E
So
during
workload
it
just
spins
up
some
of
the
resources
which
we
specify
as
a
part
of
config
file,
and
you
know,
registers
a
measurement
Watcher
on
top
of
them
just
to
kind
of
keep
track
of
those
resources
to
you
know
calculate
measurements.
On
top
of
it,
so
once
that
is
done,
you
know
it
spits
out
some
information
about
configuration
file
like
the
configured
QPS.
E
You
know
the
resources
which
are
being
deployed
as
well
as
the
preload
step
going
on
here
once
that
is
done,
it
cleans
a
preload
namespace
and
then
you
know,
starts
creating
all
the
resources
which
were
designed
as
a
part
of
workload.
For
example,
if
you
look
at
this
iteration,
so
it
has,
you
know
to
deploy
three
deployments
created
here
and
also
you
should
be
able
to
spot
some
of
the
secrets,
which
were
also
specified
as
a
part
of
configuration
file,
so
this
uses
multi-processing
to
create
all
those
resources.
E
This
kind
of
creates
stress
on
top
of
the
cluster,
we're
creating
all
those
resources
and,
at
the
end
it
makes
sure
all
those
resources
are
reaching
to
the
expected.
You
know
end
state,
which
is
in
our
case
ready
straight
for
the
pods.
So
once
that
is
done,
you
know
that's
what
this
parameter
of
wait
when
finished
true
means
to
check,
for
you
know,
resources
if
they
are
in
the
ready
State
at
the
end,
so
once
it
reaches
that
state,
it
verifies
the
count
of
objects
if
it
is
suspected
or
not.
E
If
it
is
not,
it
throws
error
in
the
command
line.
With
this
status
you
know
non-zero
and
once
it
verifies
all
the
resources
are
done.
It
stops
the
measurement
Factory
and
then
you
know
calculates
latencies
here
so
here
he
here
is
what
it
spits
out:
the
power
latencies
and
we
have
the
same
set
of
metrics
indexed
to
our.
You
know
desired
indexer
as
well,
so
these
are
some
of
the
metrics
like
you
know
how
much
time
on
an
average
pods
took
for
in
select
to
reach
initialized
State
scheduled
containers,
ready
and
registered.
E
E
So
these
are
the
metrics
which
are
published
in
our
local
file
system,
so
these
were
these
are
with
the
names
which
are
specified
in
The
Matrix
profile,
for
example,
if
you
want,
if
someone
wants
to
look
at
this
99th
hcd
square,
if
sync
duration
seconds,
they
can
simply
open
the
open
up.
This
Json
and
you
know,
look
at
the
specific
metric,
and
this
is
how.
E
E
D
It's
it's
important
to
point
out
that
Vishnu
in
this
specific
demo
was
just
as
we
built
through
the
demos
he
was
just
pointing
out
the
local
indexing
option.
What
we
usually
do
for
our
work
is
use
the
remote
open
search
feature
where
we
send
all
of
these
metrics
that
were
scraped
to
an
open
search
instance
and
actually
visualize
them
through
a
front-end
grafana.
That's
our
usual
workflow
as
a
performance,
engineer
and
I.
Think
Raul's
demo
will
touch
on
some
of
that.
Yeah.
E
B
D
B
B
So
the
ocp
wrapper,
basically,
is
a
new
scene
wrapper
on
top
of
couponer
that
basically
automates
some
stuff
you
will
have
to
make
manually
in
running
algorithmetics
cluster.
These
kind
of
things
are,
for
example,
obtaining
a
Prometheus.
We
are
available
and
also,
and
also
discovering
the
Prometheus
endpoint
of
the
cluster.
So
you
don't
have
you
don't
need
to
expect
to
specified
when
you
can
see
when
you
execute
your
burner?
B
Also,
apart
from
that,
let
me
explain:
the
ocp
wrapper
is
just
visible,
with
Q
partner
space
ocp
help.
This
rapidly
includes
some
well-known
workloads.
We
with
the
performance
and
scale
team
use
in
your
day-to-day
work
basis,
so
just
to
I'm
going
to
go
through
some
of
these
workloads.
So
we
have
the
cluster
density
workloads,
which
are
basically
workloads
containing
up
the
variety
of
objects
like
deployment,
CMT
image,
streams,
builds
services
and
even
routes,
and
so
and
the
goal
of
this
workload
is
basically
to
express
the
API
and
all
on
the
sdn
components.
B
B
B
odds
per
note
and
for
data
for
that
testing.
He
he
used
a
no
density
heavy
workload
intensively,
so
I
think
it
could
be.
This
could
be
a
good
resource
if
you
wanted
to
try
this,
that's
it
by
by
yourself
in
your
clone
cluster,
so
about
how
the
ocp
rocket
works
and
show
you
how
the
index
the
metrics
are
indexes
as
well.
B
So
the
kubernet
basically
takes
care
of
the
mass
of
calculating
how
many
parts
are
currently
divided
in
the
worker
nodes
and
determining
how
many
Bots
has
to
create.
So
if
I
hit
enter
the
Party
begins.
Thank
you.
Vulnerable
start
doing.
Stuff
of
the
first
moment
is
taking
some
time
because
it
is
collecting
some
metadata
that
is,
will
that
will
index
along
with
the
metrics
as
usual,
it
is
going
to
bring
some
of
the
configuration
parameters
like
ups
and
burst
is
going
to
specify
the
number
of
iterations
it's
going
to
execute.
B
Well,
it
also
presents
some
quick
summary
about
the
about
the
sorry
about
the
the
latest
symmetrics
observed.
So
we
can
see
here.
Let
me
go
through,
and
so
we
can't
see
that
the
both
Reggie
latency
99
percentile
was
was
50
15
seconds
and
the
average
was
8
500
milliseconds
of
proximally,
and
also
it
specifies
the
latency
for
another-
is
for
the
other
life
cycle.
Events
of
the
post,
like
initialize
container,
variable
to
schedule
it.
You
have
this
life
cycle.
Events
in
the
official
kubernetes
documentation
in
case
you're
curious.
B
What
does
it
mean
exactly
and
once
it
finishes
showing
all
this
stuff
and
it
indexes
the
metrics
and
paints
which
metric
is
basically
scrapping
from
Prometheus
and
how
much
time
is
taken
to
index?
How
much
time
is
taken
to
index
the
document
into
the
Q
Unit
13
stands
on
how
many
documents
it
has
created?
E
B
B
So
this
worker
latest
list
lasted
just
some
seconds
and,
as
you
may
know,
Prometheus
script
metrics
on
30
second
base,
30
second
basis.
So
for
that
reason,
there's
only
one
data
point
here
in
this
metrics,
which
is
not
very
useful,
as
you
can
see,
because
this
workload
only
lasted
lasted
around
27
seconds
but
as
a.
B
Can
see
that,
for
example,
we
can
open
this
drop
the
row
and
see
offset
what
were
the
all
latency
Matrix
collected.
We
can
even
go
to
Port
regularity
and
see
what
was
the
latency
observed
in
every
port
created
during
the
workload
and
well
so
we
have
included
some
or
some
other
useful
metrics
like
Master
CPU
utilization
memory
station,
that
is
to
say,
I
mean
we
are
pretty
in
this
metrics
here
because
that's
useful
for
our
day-to-day
work.
But
a
user
should
be
able
to
to
customize
whatever
he
needs
to
collect
the
metrics.
D
Of-
and
this
is
a
grafana
instance
in
our
lab-
anybody
is
free
to
send
it
to
any
era.
For
instance,
well,
grafana
is
in
the
lab
in
the
open
search
instances
in
the
cloud
right
now,
I
guess,
but
anyway,
the
point
I'm
trying
to
make
is
you
can
send
it
to
your
own
data
store.
You
don't
need
to
send
it
to
our
data
store
so
you're.
D
You
know
you're
flexible
enough
to
do
that
analysis
yourself,
but
the
important
thing
is:
if
you
want
to
do
a
regression
analysis
and
compare
run
to
run,
we
do
want
the
data
to
be
stored
somewhere.
So
that's
why
we
ship
it
to
open
search
or
elasticsearch
and
do
the
comparisons
and
Analysis
on
grafana.
So
that's
the
usual
workflow
of
my
team,
so
you'd
see
like
Raul
or
Vishnu
spending
a
lot
of
time
on
these
graphs,
analyzing
stuff
and
making
sure
that
openshift
is
scalable
and
performant
and
meets
our
customers
requirements
exactly.
A
B
C
Had
to
drop
So
for
anybody
who
noticed
that
Johnny's
no
longer
here
he's
over
I
believe
he's
helping
with
the
openshift
common
stuff,
where
they're
going
to
be
talking
about
validated
patterns
and
all
that
so
yeah,
we
I
guess
just
a
few
more
minutes.
You
know
we'll
keep
on
this
topic,
we'll
close
out
a
couple
of
things
So
for
anybody
who
is
watching
us
live,
please
feel
free
to
you
know,
put
in
any
questions
that
you
have
into
chat.
We'll
do
our
best
to
get
those
answered.
D
Now
I
was
just
gonna
say
that
was
what
we
had
planned
for
demos
we
have.
We
could
we
could
take
some
questions
or
we
there's
there's
a
couple
of
other
topics
we
could
hit
on.
But
the
entire
set
of
demos
is
on
GitHub,
but
I
will
put
them
together
and
I.
Think
that
was
shared
on
chat,
so
feel
free
to
explore
and
play
with
Cube
one
or
yourselves,
and
you
know
you
can
engage
with
us
on
issues.
D
You
know
just
on
GitHub
like
any
open
source
project.
So
we
have
a
very
open
model
for
development
and
we
don't
do
any
separate
tracking
internally
of
features.
So
everything
is
taken
from
GitHub
issues
so
feel
free
to
post
there,
and
if
you
find
any
issues,
let
us
know
and
we'll
be
happy
to
fix
them.
B
E
On
top
of
it,
like
the
triple
itself,
has
the
documentation
link
from
which
someone
can
you
know,
navigate
to
the
documentation
and
explore,
for
you
know
a
lot,
many
other
options
as
well
with
this
demo,
like
we
only
like
due
to
the
time
constraint,
we
only
had
you
know
a
restricted
functionality
that
we
could
demonstrate,
but
there
are
a
lot.
Many
other
features
which
someone
can
explore
and
bring
a
player
yeah.
D
C
D
Operators
is
controllers
plus
crds
right,
so
we
can
approach
it
from
the
lens
of
crds
I.
Think
we
had
some
very
recent
testing
to
even
take
up
the
number
of
crds
to
a
thousand
plus.
We
don't
think
we
have
updated.
The
dots
I'll
have
to
Circle
back
on
that
one.
But
again
what
we
test
is
not
what
is
the
absolute
maximum
right?
Kubernetes
scalability
is
in
multiple
Dimensions.
If
you
want
to
go
further
in
One
Direction,
you
may
need
to
compromise
on
some
other
dimensions,
so
always
keep
that
in
mind.
D
I
think
I
answered
that
on
chat,
but
I'll
reiterate
if
you're
a
performance
in
scale
engineering
team
we're
not
software
developers
per
se,
but
we
do
our
best.
We
can
and
to
make
this
a
supported.
Offering
means
we
need
a
dedicated
software
engineering
effort
which
we
don't
have
bandwidth
for
right
now,
so
our
model
is
your
mileage
might
vary
and
we
have
numerous
instances
of
customers
running
Cube
burner
even
in
their
environments
and
sending
us
data
to
our
elasticsearch
right.
D
C
I
always
tell
folks,
if,
if
that
is
something
you're
interested
in
work
with
your
account
team
and
ask
them
to
open
you
know
an
rfe
or
or
open
an
issue
with
you
know,
internally
with
red
hat,
and
we
can
associate
customer
data
with
that
and
effectively
the
more
customers
we
have
that
are
interested
the
easier
it
is
to
justify
those
resources.
As
I
said,
you
know,
get
get
resources
assigned
to
be
able
to
do
all
those
things
so
definitely
work
with
your
account
team.
C
D
Yes,
sir
Raul
went
over
the
cluster
density
scenario
and
one
of
the
features
that
it
has
is
a
churn
feature
which
is
to
mimic
a
real
cluster.
Instead
of
just
loading
up
in
an
instant
and
unloading,
an
instant
it
churns,
a
certain
percentage
of
namespaces
created,
creates,
deletes,
creates
deletes
for
a
set
amount
of
time,
so
yeah
I,
guess
that's
what
you're
asking
for.
So
it
is
yes,
yeah.
C
So
one
thing
I
wanted
to
ask
is
so
qburner
and
I
pointed
this
out
up
front.
It's
a
framework
right.
It's
an
orchestrator
for
deploying
workload,
it's
not
explicitly
for
application
performance
testing.
Rather,
it's
focused
on
the
kubernetes
aspect
and
how
kubernetes
is
being
affected,
which
is
not
to
say
you
couldn't
use
it
to
test
an
application.
You
know
and
scale
that
application
up
and
see
what
happens
and
all
that
other
stuff.
It's
just
that's
not
the
primary
use
case.
I
think.
D
Right
I'll
give
you
a
good
example
here
right.
So
Andrew
Collins
is
an
engineer
on
my
team
that
worked
on
the
2500
Parts,
Bernard
blog,
which
was
shared
in
the
chat
recently
and
one
of
the
workloads
he
used
to
do.
Drive
the
entire
exercise
was
called
No
density,
heavy,
which
is
an
actual
scenario
within
Cube
burner
that
I
believe
Raul
put
together
back
in
the
day.
D
What
it
has
is
a
pod
that
an
application
pod
that
does
some
computation
and
comes
up
with
a
Euler
number
or
something
and
the
postgres
part
right,
which
is
fronted
by
a
service
and
the
application
part
just
puts
that
data
into
that
postgres
pod.
So
we
do
actually
run
applications
through
Cube
burner
but,
like
you
said,
the
intent
being,
what
does
the
application
do
to
the
platform?
And
how
can
we
support
the
application?
We're
not
really
concerned
about?
D
B
Sense
yeah,
as
I
said
previously,
is
it
possible
for
to
specify
your
partner
to
scrape
the
user
workload
monitoring
stack?
So,
for
example,
you
have
some
custom
application,
exposing
some
math
quick
standpoint.
You
should
be
able
also
to
collect
the
metrics
from
your
application
in
case
you
want
to
test
it
massively
in
your
allowed
to
scaling
your
cluster.
E
And
also
like
when
it
comes
to
application
testing
just
to
add,
on
top
of
like
what
size
it,
we
already
have.
No
density
heavy,
which
you
know
kind
of,
creates
a
simple
app
and
you
know
scales
it
up
to
create
the
performance
and
load
test,
but
also
like
a
user,
can
specify
a
customer
application,
because
requirements
vary
like
some
people
wanted
to
test
a
Java
application.
Some
people
wanted
to
test
a
you
know,
low
level
like
C
compound
application.
E
They
can
just
you,
know,
containerize
it
and
then
specify
it
as
a
object
in
their
configuration
and
then
specify
iterations
on
top
of
it,
so
that
they
can
have
their
have
the
minimal
configurations
specified
over
there,
which
will
scale
up
to
the
bigger
size
of
the
application,
and
then
they
can
collect
metrics
on
top
of
it.
So
that's
the
main
flexibility
aspect
of
kubernet,
wherein
you
know
everyone
can
specify
you
know,
workloads
and
specify
the
applications
according
to
their
requirements
and
do.
C
C
Yeah
it
so
I
have
a
I,
have
a
whole
Bevy
of
questions.
So
is
this
something
that
you
would
recommend,
or
you
think
would
be
appropriate
to
kind
of
have
continuously
running
in
the
background
of
a
cluster,
you
know
kind
of
keeping
up
with
the
you
know:
hey
I
want
to
know
if
something
changes
inside
of
my
cluster,
you
know:
I
I've
got
these
namespaces
that
are
running
my
production
workload
and
these
namespaces,
where
co-burner
is
you
know
just
banging
up
against
you
know
the
API
server
and
SED
and
all
that
other
stuff.
D
Looks
like
the
curveball
is
for
me,
then,
because
I
will
Envision
are
silent,
so
there
are
multiple
facets
too
Cubone
right
the
indexing,
the
major
part
of
it
where
you
can
tell
Cuban.
If
this
is
the
start
time.
This
is
the
end
time
go
and
get
metrics
for
that.
I
can
definitely
see
that
being
useful
for
analysis,
but
I
wouldn't
run
Cube
one
or
continuously
on
a
production.
D
E
C
C
E
Test
continuously,
it's
not
similar
to
a
continuous
profiling
tool,
which
you
know
continuously
monitors
on
top
of
the
cluster,
so
it
is
something
which
creates
load
and
it
I
I.
Don't
think
like
this
will
be
useful
to
you
know,
detect
regressions
on
the
continuous
time
period
of
time
like,
instead
of
triggering
and
using
a
load
test
tool
like
I
would
recommend
to
go
with
the
monitoring
tool.
You
know
in
order
to
you,
know,
detect
regulations
and
spikes
over
a
period
of
time.
C
D
That
reminds
me
of
something
very
funny:
Andrew
I
remember
seeing
a
customer
busy
open
and
it
was
not
the
customer
scenario
being
explained
as
to
how
they
hit
the
bug
right
or
the
issue
it
was
like.
Hey,
I
ran
Cube
burner
with
these
parameters,
and
you
know
I
couldn't
scale
past
so,
and
so
so
we
even
have
instances
of
customers
opening
bugs
based
on
Cube
burner
again,
like
we'll
do
our
best
to
support
the
customers.
D
E
Also
good
thing
about
kubernet
is
like
if
you
already
have
a
workload
run,
which
is
in
a
previous
timestamp
or
a
like.
You
know,
or
a
previous
day
you
can
just
Mark
those
timestamps
and
feed
it
as
an
input
to
cube
burner
which
will
just
scrape
the
metrics
at
that
point
in
time
like
to
understand
the
behavior
of
the
cluster
on
that
point
in
time,
without
running
any
actual
workload.
C
Yeah
yeah
and
our
hope,
nine
makes
an
interesting
comment
here
of
you
know
using
it
to
periodically
Baseline
the
cluster
just
to
check.
For
you
know
it
changes.
You
know
Improvement
or
decline
over
time
that
type
of
stuff
in
order
to
to
verify
I,
did
have
a
a
thought
of
you
know:
I
see
kind
of
two
important
use
cases
right,
so
one
of
course,
is
baselining
and
just
verifying
validating
that
you
know
on
day.
One
I
can
expect
this
type
of
performance
you
know
or
or
this
amount
of
scalability
inside
of
the
cluster.
C
The
other
side
is
I.
Think
using
that
Baseline
in
order
to
determine
whether
or
not
changes
are
going
to
have
a
positive
or
A
negative
effect,
and
this
is
something
that
I
recommended
all
the
time.
When
I
was
a
storage
admin
right,
hey
we're
going
to
go
in
and
we're
going
to
tweak
this
little
setting
and
see
what
happens?
You
know
I,
see
Suresh.
You
know
mentioning
iSCSI
there
right,
hey
we're
going
to
change
the
on
tap
iSCSI
service
and
we're
going
to
set
this
parameter
well
great.
Did
it
actually
do
anything?
C
Did
it
help
with
what
you're
doing
so
you
know
those
things
of
you
know,
I
see
periodically
threads
come
through
where
people
want
to
go
in
and
they
want
to
tweak
SCD
settings,
and
you
know
change
leader,
election
timeouts
and
all
this
other
stuff
and
it's
one
of
those
like
okay.
You
know
what
are
you
trying
to
fix?
Is
that
really
going
to
fix
it
and
oh,
let's
actually,
test
and
validate
that
it's
going
to
positively
impact
or
result
in
the
positive
behavior
that
you
want
so
I
think
that's
super
important.
B
C
B
Example,
this
is
common
thing.
We
have
found
out
very
many
times.
Our
customer,
for
example,
stating
that
it
has
upgraded
a
storage
layer
to,
for
example,
include
a
nvmi
devices
instead
of
the
regular
SSD
and
for
LCD
I'm.
The
customer
in
this
case
wanted
to
verify
what
is
the
impact
of
having
those
nvme
devices
in
a
real
scenario,
rather
than
doing
synthetic
tests
or
with
fio
or
using
the
old
tools
by
ATD.
B
C
B
Yes,
for
example,
for
the
northern
city
test.
We
internally
have
to
find
an
SLO
of
five
seconds
latency,
so
every
test
that
is
supposed
to
go
beyond
that
five
seconds,
latency
SLO,
we
consider
it
a
failure
or
a
regression,
so
it
has
turned
out
to
be
quite
useful
because
that
latency
is
an
indicator
that
something
not
going
well,
that
could
be
on
the
North
side,
the
cubelet,
the
container
runtime
or
even
in
the
cni
side.
So
yes,.
C
C
It's
really
interesting
to
me
to
one
see
how
the
sausage
is
made
so
to
speak,
and
it
gives
me
a
lot
of
not
that
I
didn't
already
know
this,
but
it
gives
me
a
lot
of
confidence
and
a
lot
of
you
know
just
reassurance
to
know
that
you
all
are
working
diligently
behind
the
scenes.
You
know
I
said
before
that
I've
seen
your
presentations
and
I
love
those
presentations
that
walk
through
you.
C
We
tested,
you
know
sdn
performance,
we
tested
API
server
performance,
we
tested
all
these
different
things
and
I
think
that's.
That's
really
awesome
so
to
our
audience.
So
for
anybody
who
is
watching
us
live.
You
know
we'll
be
here
for
another
60
seconds
or
so
feel
free
to
drop
in
any
questions.
We'll
do
our
best
to
answer
them
or
we
will
follow
up
next
week
during
the
top
of
Mind
segment.
C
C
No
that
one
didn't
work
come
on
well,
I
thought
I
had
a
chat
shortcut
that
would
actually
post
a
link
into
the
kubernetes
slack,
but
it
doesn't
seem
to
be
working
so
instead
I'll
go
with
our
normal
caption,
so
you
can
always
reach
out
to
us
on
the
kubernetes
slack,
so
kubernetes.slack.com
archive
slash,
you
see
the
URL
there
I'm
not
going
to
go
through
all
of
that
thing.
One
thing
if
you
are
not
already
a
member
over
there
just
to
show
you
what
that
looks.
C
Like
so,
if
I
go
here,
because
somebody
sent
us
a
message
and
asked
us
the
other
week,
so
if
I
go
to
slack
dot,
oops
kubernetes
dot
slack.com
up
here,
if
you
don't
already
have
an
account
that
you
can
log
in
with
you,
click
create
an
account
and
it
takes
you
to
this
page.
You
see
this
thing
right
here.
It
tells
you
exactly
what
you
need
to
do
in
order
to
get
an
account,
so
it
says,
get
an
invite
at
you
know:
Slack
Dot
kubernetes.io
and
if
I
go
to
that
URL.
C
C
You
can
also
reach
out
to
us
on
openshiftcommons.slack.com
so
that
one
does
not
have
an
inviter
we're
trying
to
figure
that
out,
but
you
can
always
I
think
you
can.
If
you're
a
member
of
Commons,
you
can
ask
them
and
they
know
how
to
send
out
the
invites
from
there
and
of
course,
email
is
always
great
andrew.sullivan
redhat.com
or
johnnyjo
NY
redhat.com
and
with
all
that
being
said,
thank
you
really
appreciate
you
all
joining
us.
Today.
It's
been
great
having
you
on
and
I
left
Johnny's
name
up
there.
D
D
C
C
C
Hope,
nine,
that
that
was
usually
my
experience
as
well.
Something
has
changed
and
it's
changed
for
the
worse.
What
did
you
do
and
oh
we
didn't
do
anything.
You
know
it's!
It's
fine,
it's
okay,
so
anyways!
All
right!
My
son
is
now
sending
me
multiple
messages.
So
all
right
chill!
Thank
you
so
much.
We
will
see
everyone
next
week.
Next
week
we
will
have
ortwin
and
stelianos
joining
us
to
talk
about
service
mesh
so
be
sure
to
join
us.
At
the
same
time
here
on
red
hat,
live
streaming
have
a
great
week.