►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Improving the Performance of Your Kubernetes Cluster - Priya Wadhwa, Google
Ever wonder if your Kubernetes cluster could be more performant? Most users of Minikube, a tool to run Kubernetes locally, have; in fact, improved performance has been one of the most requested features since the project began. In this talk, Priya Wadhwa will discuss how we tackled reducing the start latency and CPU overhead of a Kubernetes cluster. You will walk away with knowledge of frameworks and tools to identify where performance issues exist in your own cluster. We’ll cover how we used these tools to identify where overhead was coming from and how we came up with solutions to reduce it, all while maintaining a great user experience.
https://sched.co/Zerq
A
A
I've
also
maintained
other
open
source
projects
like
scaffold
and
canico
and
if
you're
interested
in
keeping
up
with
any
of
these
tools,
I
have
my
twitter
and
github
information
at
the
bottom
of
the
slide.
I
also
just
want
to
preface
this
talk
by
saying
that
this
talk
is
about
improving
kubernetes
performance
and
when
I
first
started
delving
into
this
at
the
beginning
of
the
year,
I
was
a
complete
beginner.
A
A
So
a
quick
agenda
for
this
talk.
As
I
mentioned
before,
I
was
a
complete
beginner
when
I
started
learning
about
performance
at
the
beginning
of
the
year.
So
I'm
going
to
start
by
talking
about
the
different
performance
tools
that
I
learned
how
to
use
mostly
focusing
on
linux
performance
tools
and
go
performance
tools,
and
then
I'm
going
to
talk
about
how
I
use
the
analyses
from
these
tools
to
actually
improve
kubernetes
overhead.
A
So,
as
I
mentioned
before,
I
maintain
a
tool
called
mini
cube,
which
runs
kubernetes
on
your
local
machine,
and
this
talk
is
going
to
be
a
case
study
in
how
I
improve
performance
for
mini
cube
specifically,
but
I
think
a
lot
of
the
tools
and
techniques
that
I'm
going
to
talk
about
are
applicable
to
any
kubernetes
cluster,
even
not
just
a
local
single
node
cluster.
So
yeah
stick
around
if
you're
interested
in
performance.
A
A
So
you
might
be
wondering
why
we
even
wanted
to
improve
mini
q
performance
and
the
reason
is
it
was
highly
requested
by
our
users.
I
don't
know
about
you,
but
when
I'm
developing
on
kubernetes,
especially
locally,
I'm
also
usually
working
in
an
ide
and
streaming
youtube
videos
and
responding
to
emails
all
at
the
same
time,
and
this
ends
up
taking
up
a
lot
of
cpu
and
it
seems,
like
a
lot
of
our
users
are
doing
the
same
thing
over
the
past
few
years.
A
We've
gotten
a
lot
of
issues
around
minicube
using
up
way
too
much
cpu
and
the
one
that
really
stands
out
is
the
one
on
the
bottom
left.
Vm
has
50
resting,
cpu
usage
when
idle
and
when
you
think
about
it,
that's
pretty
bad.
That
means
that
just
running
kubernetes
on
your
machine
already
costs
you
around
50
of
a
core,
and
that
number
is
only
going
to
go
up
as
you
actually
start
to
deploy
your
application
and
develop
on
it.
A
A
The
other
script
that
I
used
quite
frequently
was
written
by
another
mini
cube.
Maintainer
thomas
and
it's
called
c-stat
and
c-stat
is
basically
a
more
precise
iostat
and
it
calculates
the
overhead
of
the
entire
system
and
the
way
that
we
used
it
was.
We
would
use
csat
to
calculate
the
system,
overhead
with
mini
cube,
the
system
overhead
without
mini
cube,
and
then
we
would
subtract
those
numbers
to
figure
out
what
the
overhead
of
mini
cube
was.
A
A
A
We
realized
that
we
could
just
pause
kubernetes
in
mini
cube
when
we
didn't
need
it,
and
this
inspired
the
mini
coupe
pause
for
the
mini
cube
unpause
command.
So
the
idea
is
is
that
if
you're
running
kubernetes
on
your
local
machine
and
then
you
deploy
your
application-
and
you
realize
that,
maybe
you
don't
need
kubernetes
for
the
next
20
minutes
or
something
you
can
just
run.
A
So
as
a
beginner,
I
realized
that
the
first
thing
I
needed
to
do
was
learn.
How
to
use
performance
tools?
I
figured
if
I
could
first
analyze
kubernetes,
then
maybe
that
would
help
me
kind
of
figure
out
where
I
should
be
looking
for
potential
performance
improvements,
and
I
focus
on
linux
performance
tools
and
go
performance
tools.
So
I'm
going
to
talk
about
the
linux
performance
tools
first
and
luckily
there
is
a
lot
of
really
good
information
online.
A
So
the
use
method
by
brendan
gregg
is
basically
a
methodology
which
allows
you
to
analyze
the
components
of
any
linux
system,
and
I
thought
that
this
would
be
a
really
good
place
to
start,
because
he
basically
provides
this
really
handy
checklist,
which
lists
all
of
the
different
components.
You
might
want
to
check
things
like
cpu
or
memory
or
io.
A
B
A
The
next
thing
I
looked
at
was
were
bcc
tools,
so
ebpf
stands
for
extended
berkeley
packet
filter
and
the
recommended
front
end
to
use.
Evpf
are
bcc
tools,
and
basically
these
are.
This
is
just
a
huge
collection
of
tracing
tools
written
in
python,
which
help
you
profile
and
trace
the
linux
kernel.
So
if
you're
interested
in
running
bcc
tools
in
minicube,
I
have
the
link
in
that
third
bullet
point
on
the
slide.
A
But
the
really
cool
thing
about
this
is
the
number
of
tools
and
the
number
of
components
of
your
kernel
that
you
can
observe
and
trace.
With
these
tools.
I
took
that
diagram
from
the
readme
on
their
github
page
and
there's
just
there
were
so
many
different
commands
and
I
ended
up
experimenting
with
running
a
bunch
of
them
on
minicube,
just
to
get
a
sense
of
what
was
going
on.
A
And
one
of
those
commands
was
called
biosnoop,
which
is
a
command
to
trace
block
device
io,
and
it
includes
the
pid
of
the
caller
and
the
latency
of
the
call,
and
I
noticed
that
when
I
was
running
this
command
in
the
mini
cube
vm,
that
lcd
was
writing
to
disk
pretty
frequently.
So
keep
that
in
mind.
A
The
next
thing
I
did
was
generate
a
flame
graph
of
my
of
my
system,
so
flameclass
are
really
useful
for
figuring
out
which
code
which
functions
in
on
your
machine
are
contributing
to
overhead,
and
this
is
a
flame
graph
that
I
generated
of
my
entire
linux
system
over
the
course
of
a
minute.
So
if
you
look
at
the
x-axis,
you
can
see
stack
profile,
population
ordered
alphabetically
and
the
wider
that
a
frame
is
the
more
often
that
code
path
came
up.
A
So
if
you
look
at
the
x-axis,
you
can
see
that
two
cpus
are
assigned
to
kvm
and
they're
contributing
a
bunch
to
overhead
chrome
is
contributing
a
bunch.
Something
called
swapper
is
contributing
a
lot
as
well.
But
since
I
know
that
minicube
is
running
in
a
vm
and
in
this
case
I
know
that
mini
cube
is
running
in
kvm
and
I've
assigned
two
cpus
to
it.
I
knew
that
the
two
stacks
on
the
far
left
were
coming
from
kubernetes.
A
So
this
is
the
zoomed
in
stack
for
one
of
the
cpus
that
was
assigned
to
minicube
and
running
kubernetes,
and
I
noticed
that
a
syscall
called
ioctl
was
taking
up
like
a
huge
chunk
of
the
stack
and
was
like
pretty
much
like.
It
looks
like
it's
almost
around
75
of
the
entire
frame
width,
and
so
I
was
curious.
I
was
like,
oh,
what
is
it
ctl
and
I
didn't
know
so
I
googled
it.
A
I
don't
know
about
you,
but
I
really
think
that,
like
75
of
learning
about
kubernetes
and
performance
engineering
for
me
has
been
googling
things
and
unsurprisingly,
iocto
has
to
do
with
io
operations,
and
this
kind
of
reminded
me
of
all
the
fct
rights
that
I
had
seen
when
I
was
running
biosnu
and
in
my
use
analysis
as
well.
I
had
noticed
that
xcd
was
writing
to
disk
a
lot
when
I
had
run
iotop
as
one
of
the
commands.
A
So
I
thought
that
if
I
increase
the
value
of
snapshot
count,
those
backups
would
be
saved
to
disc
less
frequently
and
maybe
require
less
overhead.
So
I
tried
tuning
this
value
from
values
of
500
all
the
way
up
to
a
hundred
thousand,
and
unfortunately,
I
like
didn't
see
any
performance
improvement.
In
fact,
for
almost
every
single
value
overhead
actually
got
worse
and
except
for
one
case,
one
snapshot.
Count
was
three
thousand,
but
in
that
case
there
was
only
two
percent
improvement,
so
it
didn't
really
seem
statistically
significant.
A
I
basically
learned
how
to
use
flame
graphs
and
vcc
tools
and
all-
and
I
created
a
use
analysis
to
kind
of
have
a
general
understanding
of
the
performance
of
mini
cube
and
of
kubernetes
and
from
all
these
things,
I
noticed
that
scd
was
writing
to
disk
a
lot
and
that
it
seemed
to
be
contributing
to
overhead.
So
I
tried
tuning
the
snapshot
count
flag
to
see
if
it
would
reduce
the
number
of
scd
rights
and
there
was
no
performance
improvement
at
all.
So
I'm
not
gonna
lie
at
this
point.
A
And
so
I
ran
this
command
called
hidstat
160,
which
had
also
been
part
of
my
use
analysis,
and
I
noticed
that
every
few
seconds
there
was
a
crew
control
command
that
was
using
a
ton
of
cpu.
So
you
can
see
that
they're
kind
of
in
bold
that
these
two
control
commands
were
using
nine
percent
of
a
core
and
four
percent
of
a
core,
and
this
was
significantly
more
than
all
of
the
other
processes
that
we're
running.
A
So
I
appended
the
dash
l
flag
onto
the
command
just
to
see
where
it
was
coming
from,
and
I
noticed
that
the
food
control
command
was
control
apply
and
it
was
being
applied
to
the
kubernetes
add-ons
directory,
and
so
I
had
a
feeling
that
the
add-on
manager
was
responsible
for
running
this
command.
And
the
other
thing
I
noticed
is
that
this
coupe
control
apply
command
was
exactly
the
same
being
run
every
few
seconds
as
if
it
was
polling.
A
A
The
idea
is
once
you
spin
up
minicube,
you
have
kind
of
a
standard,
kubernetes
cluster,
but
maybe
you
need
some
extra
things
in
your
cluster
like
helm
or
tiller
or
something
else,
and
so
you
can
run
mini,
cube,
add-ons
enable
helm
tiller
and
what
will
happen
once
you
do?
That
is
that
minicube
will
copy
the
required
kubernetes
manifest
for
help
for
tiller
into
the
kubernetes
add-ons
directory
and
then
eventually
add
the
add-on
manager
will
run.
Kube
control
apply
on
that
directory
and
the
required
pause
will
pop
up.
A
A
So
at
this
point
it
became
pretty
clear
that
the
add-on
manager
was
creating
these
spikes
and
a
really
easy
solution
presented
itself.
I
could
just
increase
pull
time
and
the
higher
pull
time
was
the
lower
average
overhead
would
become,
but
unfortunately
this
would
create
a
trade-off
between
overhead
and
user
experience,
because
if
the
add-on
manager
is
only
polling
every
two
minutes
for
changes
in
the
directory,
it
could
take
users
up
to
two
minutes
to
see
add-ons
changes
in
their
cluster.
A
Luckily,
in
this
case,
I
didn't
end
up
having
to
choose
between
the
two.
I
realized
that
we
could
remove
the
add-on
manager
entirely
because,
as
many
cube
users,
we
know
when
users
are
making
changes
to
the
state
of
add-ons
in
the
cluster.
They
have
to
physically
run
mini
cube,
add-ons
enable
or
mini
cube
add-ons
disable,
and
I
realized
that
we
could
just
run
the
cube.
Control
apply
command
ourselves
whenever
a
user
said
they
wanted
to
make
changes
to
the
add-ons.
A
A
This
is
a
meme
summary
of
what
we
did.
It's
basically
what
I
just
said.
Instead
of
increasing
add-on
management
pulling,
we
just
removed
it
entirely
and
rewrote
a
version
that
was
better
seated
for
what
we
needed,
and
this
actually
resulted
in
a
32
reduction
in
overhead,
which
was
really
exciting
because
we
pretty
much
met
our
goal
just
by
removing
one
thing.
But
at
this
point
we
kind
of
wanted
to
see
like
oh,
how
much
further
can
we
take
this
like?
A
Are
there
more
improvements
that
can
be
made,
so
the
next
thing
I
needed
to
figure
out
was
how
each
part
of
kubernetes
is
contributing
to
overhead.
I
figured
that,
if
I
could
figure
out
if
I
could
like
isolate
the
different
components
and
figure
out
what
was
the
biggest
contributor,
that
would
be
a
good
place
to
start
when
trying
to
figure
out
where
I
should
start
proving
something.
A
So
you
can
see
on
from
this
pit
stat
output,
which,
by
the
way,
pistat,
was
probably
my
favorite
command
throughout
this
entire
journey.
I
learned
a
lot
from
it,
but
you
can
see
that
coupe
api
server
is
using
up
the
most
amount
of
cpu.
So
I
thought
that
api
server
would
be
a
good
place
to
start
looking
for
performance
improvements,
and
the
other
thing
of
interest
is
that
there
are
two
processes
called
core
dns
and
even
though
they're
not
using
a
whole
lot
of
cpu.
A
There
are
two
of
them
and
that's
because
the
core
dns
deployment
by
default
spins
up
two
replicas.
I
realized
that
in
a
local
single
node
cluster,
this
probably
wasn't
necessary,
like
a
regular
user,
probably
only
needs
one
instance
of
core
dns
for
their
application
to
work
successfully,
and
so
I
decided
that
scaling
it
down
to
one
by
default
would
be
a
pretty
easy
way
to
reduce
overhead
and
not
have
to
lose
anything
in
terms
of
user
experience.
A
Well,
let's
move
on
to
the
onto
the
api
server
and
how
we
improved
overhead
there
before
we
do
that.
I
want
to
talk
a
little
bit
about
pprof.
So
peop
is
a
go
tool
for
visualizing
and
analyzing
profiling
data
and
the
way
that
it
works.
Is
you
basically
import
pprof?
As
a
library
into
your
go
application
and
then
when
you're
as
you're
running
your
application,
every
millisecond
pprof
will
take
a
stacked
trace
of
your
application
as
it's
running
and
then
it'll
kind
of
collate.
A
A
So
the
cool
thing
about
ppop
is
that
you
can
represent
the
same
data
as
a
flame
graph
as
well
and
thomas
noticed
that
the
api
server
was
spending
a
lot
of
time
dealing
with
incoming
requests,
and
so
he
wondered.
Where
are
these
incoming
requests
coming
from?
Because
in
theory,
if
you
can
reduce
the
number
of
requests,
then
maybe
you
can
reduce
overhead
so
to
figure
out
where
the
requests
were
coming
from.
A
We
basically
set
verbosity
on
the
api
server
to
10,
so
that
every
request
would
be
would
show
up
in
the
logs,
and
then
we
wrote
some
bash
to
figure
out
where
the
requests
were
coming
from,
so
that
second
command
basically
pulls
out
the
user
agent
from
every
get
request
and
then
sorts
them
and
counts
them.
And
you
can
see
that
tons
of
requests
were
coming
from
cube
controller
manager
and
cube
scheduler,
and
all
of
these
were
around
something
called
leader.
Election.
A
So
what
is
leader
election?
So
I
looked
it
up
in
the
kubernetes
documentation
once
we
found
this,
but
basically
leader
election
guarantees
that
only
one
instance
of
keep
scheduler
or
cube
controller
manager
is
making
decisions
at
any
given
time,
and
this
makes
sense
if
you
have
like
a
big
cluster
with
lots
of
nodes-
and
you
have
lots
of
instances
of
these
things
running.
A
But
we
realized
that
minicube
is
by
default,
a
single
node
cluster
and
we
only
ever
expect
to
have
one
instance
of
cube
schedule
or
cube
controller
manager
running
anyways,
so
leader
election
is
not
really
necessary
for
our
use
case.
So
we
wanted
to
know
if
we
could
turn
your
election
off
and
luckily
it's
actually
really
easy
to
turn
it
off.
A
There's
a
flag
on
the
controller
manager
and
on
the
scheduler,
and
so
we
were
able
to
set
it
equal
to
false
by
default
in
mini
cube,
and
this
resulted
in
an
18
reduction
in
overhead,
which
is
really
exciting
just
from
turning
leader
election
requests
off
and
from
reducing
core
dns
replicas.
To
one-
and
this
is
my
meme
summary-
we
basically
just
said
leo
liked-
equals
false
and
saw
some
really
great
improvements
which
was
really
nice.
A
So
this
is
the
peep
off
graph.
This
is
the
entire
thing,
which
is
obviously
pretty
big,
but
I've
cut
and
pasted
the
relevant
bits,
and
you
can
see
that
the
forty
percent
from
this
call.syscall
actually
originated
with
a
function
in
the
http
library
called
the
right
frame
async.
So
I
kind
of
started
looking
through
sgd
code
to
see
where
right
from
right,
where
the
http
library
was
being
called
to
see.
If
maybe
that
was
something
I
could
tune
it
kind
of
like
the
api
server.
A
I
was
thinking
that
if
I
could
reduce
the
number
of
http
requests,
then
maybe
I
could
also
reduce
overhead
so
searching
through
the
code.
I
was
looking
for
calls
to
go's
http
library,
and
I
found
a
package
called
http
proxy
and
in
that
package
I
found
a
function
called
new
handler
which
basically
proxies
requests
to
the
fcd
cluster
and
periodically
updates
its
view
of
the
cluster.
A
I
noticed
that
there
was
this
argument
passed
into
the
function
called
refresh
interval,
which
is
by
default
30
seconds
and
can
be
set
by
the
proxy
refresh
interval
flag.
So
if
I
lost
you
somewhere
in
this
slide,
no
worries.
The
dldr
is
basically
that
I
had
this
theory
that
proxy
refresh
interval
was
controlling
how
many
http
requests
were
being
made
of
ncd,
and
I
thought
that
if
I
increased
the
value
of
this
flag,
maybe
the
number
of
requests
would
go
down
or
would
be
more
spread
out
and
so
overhead
would
go
down
as
well.
A
Every
value,
as
obviously
refresh
interval
got
bigger,
resulted
in
a
decrease
in
overhead
with
a
clear
dip
at
70
000.,
and
so
I
wanted
to
my
theory
was:
oh,
maybe
I
can
just
set
proxy
refresh
interval
to
70
000
and
enjoy
decreased
overhead,
but
of
course
I
did
not
want
to.
I
was
worried
that
user
experience
would
suffer
and
I
honestly
like
didn't,
really
know
what
the
repercussions
of
increasing
the
value
of
this
flag
would
be.
A
A
If
you
increase
the
time,
it
will
take
longer
for
any
endpoints
to
be
proxied
properly,
which
it
wasn't
exactly
what
I
thought
this
flag
did
when
I
first
started
tuning
it,
but
it
was
really
helpful
to
know-
and
at
this
point
I
just
wanted
to
make
sure
that
setting
it
to
70
000,
wouldn't
actually
make
user
experience
worse.
But
I
had
no
way
of
knowing
how
badly
it
would
actually
affect
user
experience.
A
So
I
decided
to
do
a
little
test
and
basically
test
the
time
from
starting
miniqueue
cluster,
to
deploying
an
application
to
being
able
to
visualize
that
application
by
a
service
url-
and
I
tested
it
with
a
proxy
refresh-
interval
equals
30
000,
which
is
the
default
and
70
000,
which
is
the
value
I
wanted
to
change
it
to
and
in
the
end
it
actually
didn't
seem
to
impact
user
experience
at
all.
Seventy
thousand
actually
had
actually
ended
up
taking
slightly
less
time
it's
time
for
the
deployment
to
succeed.
A
So
we're
nearing
the
end
of
my
talk.
This
is
basically
just
a
graph
of
overhead
in
2020,
since
we
began
trying
to
improve
overhead,
you
can
see
that
we
ended
up
actually
reducing
overhead
by
around
52
and
that
removing
the
add-on
manager
and
turning
off
leader
election
were
the
biggest
contributors
to
that.
A
So
some
takeaways
from
trying
to
improve
performance
for
the
past
eight
months.
I'd
say
the
biggest
one
is
definitely
that
removing
unnecessary
work
is
really
great,
and
we
saw
this
with
when
we
removed
the
add-on
manager,
removing
a
core
dns
pod
that
we
didn't
need
and
removing
leader
election,
which
we
also
didn't
need.
A
Removing
that
unnecessary
work
is
a
really
great
way
to
reduce
overhead,
because
you
also
don't
risk
losing
anything
in
terms
of
user
experience.
And
that
brings
me
to
my
second
point,
which
is
that
it's
important
to
consider
the
trade-off
between
overhead
and
user
experience
and
with
like
setting
proxy
refresh
interval.
It
was
really
important
to
make
sure
that
we
weren't
sacrificing
anything
just
for
a
little
for
a
small
improvement
in
overhead.
A
And
finally,
I
think
this
is
probably
just
general
to
engineering
or
maybe
anything
really,
but
collaboration
is
really
helpful
and
important.
I
never
would
have
been
able
to
figure
out
the
leader
election
equals
false
for
the
api
server.
If
I
hadn't
been
working
with
other
mini
cube,
maintainers-
and
I
wouldn't
have
been
pointed
in
the
right
direction,
unless
people
on
the
scd
slack
channel
have
been
willing
to
help
answer
my
questions.
A
So
thank
you
very
much
to
everybody
who
who
helped
along
this
process-
and
I
just
wanted
to
call
out
some
of
the
websites
online,
which
really
helped
me
learn
about
linux
performance
engineering
and
go
performance.
I
would
highly
recommend
both
of
these
blogs
to
anybody's
interested
in
either
of
these
topics
and,
if
anybody's
interested
in
related
talks
to
this
one
there's
a
deep
dive
into
mini
cube
tomorrow,
if
you're
interested
in
how
it
works
or
any
of
the
cool
new
features,
we've
been
working
on
and
there's
a
really
cool.
A
A
Hi
everyone
thanks
for
listening,
I'm
taking
live
questions
now,
so,
if
you
have
any,
please
ask
them
I'll
just
get
started
on
answering
so
a
few
people
have
asked
if
the
slides
will
be
available
and,
yes,
I
think
that
they
will
be
available
there
should
be.
I
think
somebody
responded
with
the
link
frederick
asks
in
which
version
of
minicube
these
changes
were
integrated,
we've
kind
of
been
integrating
them
as
we
found
them
throughout
the
entire
year.
A
A
Oh,
I
don't
know
if
someone
just
went
on
sorry
yeah
michael
asked
does
proc
need
to
be
run
on
each
node
or
just
against
the
api
server.
So
profit
is
run
against
a
go
application.
So
when
I
ran
it
against
the
api
server,
it
was
actually
running
against
the
api
server
code
and
you
can
run
people
on
any
application
written
and
go.
A
A
They're
kind
of
like
yeah-
I
guess
that's
like
my
that's.
My
best
answer
is
that
a
relatively
graph
was
pretty
flat
between
like
february
and
june-ish
in
2020,
and
that's
because
we
hadn't
really
found
any
performance
optimizations
in
that
time.
So
a
lot
of
these,
like
a
few
like
whether
it
was
like
a
few
percentage
points
off.
We
didn't
really
take
that
as
like
a
as
like
performance
getting
worse,
just
kind
of
as
like.
Maybe
a
slight
difference
in
the
system
that
the
numbers
were
being
collected
on
on
that.
A
Day
wallace
asks
what
do
you
think
about
micro
case
versus
kind
for
local
dev,
and
I
think
all
the
solutions
are
pretty
good.
It
kind
of
just
depends
on
your
use
case.
I
haven't
really
used
micro
case
before,
but
I've
used
kind
for
integration
testing
for
some
of
the
other
tools
that
I
work
on.
It's
really
convenient
to
like
use
it
to
spin
up
a
cluster
in
docker
and
then
be
able
to
like
run
integration
tests
against
it
easily.
A
A
A
A
A
Oh
god,
gotham
asked
20
to
40
is
a
very
wide
range
of
mini
cube
overhead.
Are
there
any
clear
patterns
which
I
think
is
probably
related
to
this
slide,
where
we
were
actually
like
measuring
mini
cube
overhead
at
the
very
beginning,
so
I
think,
like
the
range
or
like
the
spikes
here
were
caused
by
the
add-on
manager,
which
was
one
of
the
things
that
I
discussed
later
on
in
the
talk,
and
then
we
actually
have
it.
We
should
actually
probably
look
at
mini
cube
overhead
today
and
see
if
it's
stabilized
a
bit.
B
A
Have
you
tried
any
of
the
openshift
based
local
development
tools
like
crc
or
opd?
No,
I
haven't,
but
that
sounds
really
interesting.
I'll
have
to
check
those
out.
So
if
I
think
we're
coming
to
time
on
this
session,
but
if
anybody
has
any
more
questions,
please
feel
free
to
move
them
to
the
slack
channel
and
app
me
just
at
prio
and
I'm
happy
to
answer
them
there.