►
From YouTube: SIG - Performance and scale 2021-09-16
Description
Meeting Notes: https://docs.google.com/document/d/1d_b2o05FfBG37VwlC2Z1ZArnT9-_AEJoQTe7iKaQZ6I/edit#heading=h.qs7aweajr18k
A
A
A
So
we're
doing
some
testing
internally
at
pretty
large
scale
and
we're
seeing
the
vert
controller
panic
when
we
delete
a
bunch
of
vms,
not
not
a
ton
like
I
mean,
maybe
like
a
few
hundred
or
something
and
at
this
scale
there's
a
lot
of
events
that
are
occurring
and-
and
sometimes
you
know
with
this-
a
lot
of
events
occurring.
Some
of
the
edge
triggered
events
like
deleted
can
be
missed
by
fire
watch
and
the
way
that
the
controller
handles
this.
A
A
Actually,
what
ended
up
happening
is
that
we
there's
two
of
our
controllers
and
they
basically
would
alternate
between
panics
and
I,
and
it
was
hard
to
really
tell
if
it
was
kind
of
like
kind
of
the
it
sort
of
eventually
healed,
but
it's
hard
to
tell
exactly
how
it
healed
than
that.
Perhaps
there
was
enough
events
that
the
the
sort
of
deleted
key
the
this
deleted-
final
state
unknown
key,
was
eventually
flushed
and
then
tanks
were
away
it's
hard
to
really
say,
but
eventually
it
does
heal.
A
A
Okay,
there
are
even
so
this
this
sort
of
like
key
there's.
There
are
multiple
objects:
we're
actually
have
cues
for
the
other
ones,
the
other
objects
we
have
like.
Pods
and
data
volumes
have
catches
for
this.
I
think
it
was
just
virtual
machine
instances.
I
was
missing
it
and
it
occurs
in
rare
cases.
I
like
I've,
never
seen
this
before
it
just
occurred
at
this
very
at
this
kind
of
at
the
scale,
and
it
doesn't
occur
all
the
time.
A
A
Okay,
let's
go
to
the
second
one,
so
this
is
actually
from
the
same
same
incident,
same
scenario
deleting
bmis
at
large
scale
in
the
in
the
vert
controller
logs.
There
are
tons
of
these
status
reasoning,
valid
errors,
which
is
a
a
422
error
and
basically
like
the
server
or
the
request
that
you're
making
is
valid,
but
the
server
is
not
processing
it,
and
I
was
looking
around
a
little
bit
on
this
and
we
we
do
so.
A
A
My
thought
is
that
it's
it's,
this
active
pods
field,
that
something
is
failing
with
this
like
we're
trying
to
patch
it.
It's
not
working.
So
we
get
these
422's.
This
isn't
harmful
like
it's.
It's
like
it's
just,
there's,
there's
tons
of
them
in
the
log
I
mean
because
we're
cleaning
up
the
object.
It
would
be
nice
to
get
rid
of
these
and
here's
an
example
of
the
error.
A
A
I
haven't
fully
tied
it
together,
but
it's
it
only
really
showed
up
when
we
were
deleting
now.
I
didn't
notice
that
in
the
case
but-
and
it
seemed
really-
I
hadn't
noticed
this
prior
and
it
seemed
like
this.
This
only
occurred
at
large
scale.
Something
was
something
just
seemed
to
be
off.
This
is
the
because
I
mean
we've.
We've
done
like
some
other
error
code
analysis,
and
we
haven't
seen
this
many
422's.
It's
just
sort
of
there's
now,
there's
sort
of
an
explosion
of
them.
C
A
D
A
But
maybe
it's
in
the
api
server
reads
on
the
cubic
fan
server,
so
I
I
don't
know,
but
the
reason
I
posted
this
one,
those
I
think
I
have
a
little
bit
of
better
understanding
than
this
as
for
this
one,
but
I
think
there's
still
some
more
investigation
here
as
to
like
to
pinpoint
exactly
because
I
think
like
there
are
two
like.
I
know
that
we're
like
I
can
see
what
we're
we
can
see
the
air.
A
B
Yes,
so
in
our
patch
we're
doing
a
json
patch
and
we
have
a
test
condition
followed
by
the
actual,
replace
or
add
or
remove
whatever
we're
wanting
to
do
and
the
test
conditions
saying
here
is
the
way
we
think
this
struct
should
look
based
on.
What's
in
our
informer
and
what
the
information
we
have
and
here's
what
we're
wanting
to
change
if
that
test
condition
fails,
the
whole
thing
will
fail.
So
that'll
say
that
the
reality
is
that
the
thing
you're
trying
to
patch
doesn't
look
any
more
like
what
you're
thinking
it
looks
like.
B
What
could
be
happening
is
that
our
formers
are
behind
and
we're
constantly
trying
to
make
a
change
on
the
vmi
and
the
information
we're
using
to
make
that
change
is
inaccurate,
because
the
informers
haven't
caught
up
to
reality
yet,
and
I
made
an
optimization
for
the
update
path
for
the
the
vmi
controller
to
say:
don't
try
to
update
this
vmi
again
until
we've
seen
the
previous
update
arrive.
I
didn't
do
that
for
the
patch,
so
it
could
be
the
same
problem
where
so
was.
A
B
B
It's
just
manifesting
itself
differently
because
in
the
case
of
an
update,
for
example,
we're
going
to
fail
in
the
back
end,
because
the
updates
revision
or
whatever
is
not
accurate,
but
in
the
case
of
the
patch
we're
rejecting
the
update
or
the
patch,
because
the
patch
condition
is
failing
that
we've
supplied
so
we're
going
to
get
a
different
error.
I
think,
but
the
same
cause
could
be
invoked
for
both
like
it
could
be
the
same
underlying
cause
that
we're
trying
to
modify
an
object
and
the
object
that
we're
trying
to
modify.
B
A
A
Yeah
that
sounds
sounds
plausible
to
me:
okay
409,
I
said
429,
okay,
yeah,
okay,
so
yeah,
it
sounds
possible.
I
mean
so
even
like
he
said
for
additional
context,
and
this
is
sort
of
the
like.
Like
I
said
this
is
happening
at
the
same
time.
As
you
know,
the
panic
is
going
on.
So
there's
we're
restarting
for
controllers
a
lot.
Reformers
are
catching
up
a
lot
and
this
kind
of
was
kind
of
led
to
another
thing
that
that
I
was
kind
of
wanted
to
explore.
A
A
B
And
the
fix
is
relatively
simple:
we
just
have
to
follow
the
same
expectation
logic.
It's
almost
it's
only
like
maybe
20
lines
down
that
we're
doing
it
for
the
update.
We
just
need
to
follow
something
similar
for
the
patch,
but
we
need
to.
I
need
to
look
at
that
a
little
bit
just
to
make
sure
that
we're
only
issuing
the
expectation
when
the
patch
is
actually
going
to
change
something
I
I
think
that's
the
way
it
would
work,
but.
A
Anyway,
cool
okay,
yeah.
Well,
so
those
are
two
two
new
bugs
and,
like
I
said
there,
there's
some
other
ones
that
I
was
thinking
of.
Like
I
briefly
mentioned
like
the
time
it
takes
for
her
controller
to
catch
up
could
be
another
one,
but
I
want
to
get
a
little
bit
more
measurement
on
that
one
to
see
like
get
like
a
rough
amount
of
time
and
some
more
logs
to
see
just
to
get
a
better
picture
of
it
before
for
a
file
vision.
But
it's
something
that
I've
noticed.
A
A
And
I
want
to
kind
of
consolidate
here
on
a
on
a
mission:
let's
see
where's
the
first
one,
okay,
this
one
so
reduce
memory.
Overhead
of
launcher
is
the
first
one
and
the
ones
that
that
I
thought
we
had
some
overlap
on
was
the
was
this
one,
so
removing
the
monitoring
process
and
the
launcher
to
reduce
its
memory
footprint
and
then
also
the
the
profiling,
the
control
plane
high
load?
So
I
guess
so.
A
My
point
is
this
is
like
I
kind
of
wanted
to
talk
about
like
the
goal
here
and
see
if
we
can
outline
some
of
the
tasks-
and
I
think,
like
you
know,
this
is
sounds
like
to
me
like
one
of
the
tasks,
I
could
actually
go
into
here
as
a
possible
optimization
for
launcher,
but
I
kind
of
want
to
talk
about
you
know.
Some
of
the
others
so
is.
Is
it
daniel?
Is
that
who
this
user
is.
F
So
the
context
is
that
there
is
aldergy
internships
and
it
is
paid,
opens
open,
source,
community
internships
for
for
diversity
in
attack,
and
we
just
thought
that
I
mean
we
need
some
projects,
so
the
inter
internships
can
work
on
them
and
this
topic
comes
about,
or
was
there
for
a
long
time.
So
we
thought
why
not
do
it
and
we
don't
really
have
a
password.
F
We
just
know
that
we
can
probably
reduce
the
size
with
the
with
the
separate
binary
for
this
or
if
we
rewrite
it
to
different
language
right
and
the
other
option
can
be
to
remove
the
forking
all
together.
But
I
guess
that
needs
to
be
discussed.
If
we
can
guarantee
the
guarantees
that
the
forking
process
is
doing.
A
G
Yeah
exactly
look
so
I
I've
tried
to
estimate
on
this
this
removing
this,
this
beard
launcher
for
king
and
I
think,
when
I
go
gone
back
to
to
the
when
what
is
when
it
was
introduced.
There
was
there
was
some
special
case
and
that's
that
I
think
it
covers.
I
think
it's
for
container
disks.
I
haven't
really
used
this
feature
and
I'm
not
sure
if
we
ever
use
it
on
our
side.
G
So
I
don't
see
really.
I
don't
really
see
a
reason
why
we
can
just
do
it
in
in
like
special,
go
routine
or
or
something
like
that,
that
that
would
do
this
like
watching
and
yeah
like
as
daniel
mentioned
it
this
this.
This
monitoring
just
just
adds
a
lot
of
memory
and
and
to
overall
so
yeah
that
should
be
discussed.
A
D
Okay,
great,
so
I
want
to
adjust
that
the
main
purpose
of
it
is
that
if
we
have
it
as
a
precaution,
it
it
does
very
little
by
intention
with
because
that
means
that
that
it's
very
unlikely
that
this
process
crashes,
while
our
work
launcher
process
that
that
one
which
is
actually
talking
delivered
and
so
on,
and
to
do
him,
does
a
lot
a
lot
more
things.
And
if
it
crashes
our
main
process
and
the
container
is
down
and
then
it
then
all
other
processes
would
be
stopped
immediately
and
yeah.
It's
a
precaution.
A
Right,
so
we
don't
want
it
as
pit
one
right.
We
wanted
something
else
so
that
we
just
don't
immediately
once
it
fails.
We
just
go
away
like
so.
In
other
words,
we
it
makes
like,
because
I
I
understand
like
like
we
want
some
sort
of
something
else
to
to
to
be
there.
So
I
guess
the
question
is
like,
so
you
knew
you
said
created
another
go
routine
like
if
we
have.
B
Excuse
me,
sorry
if
people
crashes
today
or
I'm
sorry,
burt
launcher
pat
crashes
today,
the
point
of
the
pid
one
and
the
forker
is
that
we
do
some
graceful
cleanup
of
the
queueing
process
if
it's
still
around
so
we'll
attempt
to
shut
it
down
in
a
way.
That's
not
gonna
cause
this
corruption,
and
things
like
that.
So
I
think
maybe
that's
the
underlying
reason
why
we
even
have
a
catch-all
kind
of
thing
like
that.
Originally
it
was
a
bash
script.
B
What
you
said,
yes,
that's
different,
so
originally
we
had
a
bash
script.
That
would
do
this
kind
of
clean
up.
It
was
really
unwieldy
and
we
just
made
a
function
and
vert
launcher
that
would
do
it
and
then
we
just
decided
to
fork
launcher
from
vert
launcher.
I
think
we
can
achieve
the
same
thing
in
a
go
routine
and
have
it
catch
panics.
So
very
first
thing
we
do
when
burnt
launcher
starts,
is
we'll
create
this
go
routine,
we'll
have
it
catch
panics.
B
A
B
We
don't
have
to
fork.
I
think
that
we
can
do
it
in
a
go
routine.
When
a
panic
occurs,
we
have
the
opportunity
to
still
execute
something.
I
think
that
will
work.
Maybe
maybe
somebody
else
has
more
thoughts
on
that.
If
we
don't
do
that,
the
other
alternative
is
to
create
a
really
stripped
down
forker.
B
That
only
does
exactly
what
we
need
that
fork
logic
to
do,
rather
than
loading
everything
that
vert
launcher
needs
because
there's
just
a
lot
of
dependencies
and
everything
to
get
started
with
vert
launcher
and
if
we
just
have
a
really
really
small,
thin
binary,
that's
just
in
charge
of
launching
burnt
launcher
and
then
sharing
a
vert
launcher
exits
that
the
commuting
process
is
torn
down
as
gracefully
as
possible.
Then
that's
great
and
the
result.
B
A
B
Look
at
the
recover
command
and
go
laying.
I
think
that
we
set
a
recover
command
at
the
very
top
level
of
like
the
hierarchy.
That
will
basically
be
a
catch
all
for
everything.
That's
the
very
first
line
we
have
in
the
code.
I
would
think
that
anything
that
causes
the
second
fault
from
there
would
I
could
be
wrong,
but
that's
my
expectation
we'll
get
caught.
It's
not
mine,
but
no,
it's
not
yours.
Okay!
Maybe
I'm
totally
wrong
there
investigate
that.
I'm,
like
50
sure
now.
B
A
D
F
A
Yeah
they
don't
mind
well,
so
so
what
are
the
so
we
have
so
we
could
do
panic.
We
could.
We
could
have
something
to
cover,
I
guess
the
panics,
but
that
so
like
that.
We
said
that
doesn't
catch
everything
right
like
if
it's
like
what,
if
like,
what
are
the
other
cases
that
we
want
to
like
what,
if
qmu,
just
kind
of
like
crashes,.
D
A
A
Okay,
all
right,
let's
kind
of
bubble
up
a
little
bit
higher.
So
we
have
this
like
this
issue.
We
have,
we
have
to
produce
memory
already.
I
think
this
is
probably
one
of
the
issues
that
we
can
produce
the
memory
overhead
in
there.
So
I
mean
I
guess
what
we
could
do
is
we
can
kind
of
tag
cross-tie
the
issues
and
this
could
be
one
of
them,
but
there's
there's
probably
more.
Maybe
we
can
like.
We
have
profiling
that
we
want
to
do.
A
E
E
E
Yeah
yeah
the
first
image
now
yeah,
okay,
the
vm
creation
time.
Yes,
so
it
has
like
600
and
800
the
two
last
ones
and
the
first
one
took
like
five
minutes:
the
95
percentile.
You
know
the
worst
case
to
create
the
vm,
the
vmi
and
then
with
800.
It's
you
know
it's
jumped
to
then
to
10
minutes,
so
double
the
time
to
create
the
vmi
so
and
it
was
not.
The
double
of
number
of
yams
was
600
to
800.
So
I
would
say
is
not
scaling
very
well.
You
know
the
creation
here.
A
A
E
A
E
A
E
Yeah,
however,
you
see
that
it
doesn't
gross
too
much
when
we
have
like
more
vms
being
created.
You
know
from
600
to
800
it's
it's
small,
the
difference,
so
something
yeah.
It's
not
it's
not
because
of
the
scale.
It's
all
that
it
looks
like
it's
not
because
of
the
scale
of
the
vm
creation,
but
something
is
stuck
in
the
code.
So
that's
the
definition
of
unfinished
work,
but
from
kubernetes
this
metric
that
it's
when
it
grows
the
time
here.
It
means
that
some
threads
are
stuck
so
not
necessary.
A
E
Also
just
work,
you,
the
work,
you
add
rate
down
yeah
this.
I
think
we
already
have
an
issue
for
that,
but
it's
still
like
the
vm
controller
disruption
budget.
It's
the
most
intensive
one,
and
we
have
like
some
discussion
about
that.
You
know
previously
here
and
I
don't
know
who
who
mentioned
that
that
this
controller
is
actually
the
description
budget
shouldn't
be
that
intensive.
D
E
B
E
B
It
shouldn't
be,
you
shouldn't,
be
creating
that
much
work.
Yeah
it'd
be
interesting
to
profile
that
it's
creating
pod
destruction,
budgets
for
virtual
machines
that
have
eviction
strategy
equals
live
migrate
to
ensure
that
the
the
vmi
can't
be
torn
down.
What's
in
my
trusty
ray
node,
so
we're
forcing
it
to
the
eviction
to
fail
and
we're
creating
a
live
migrate
as
a
result.
G
B
A
A
A
C
E
A
A
C
I
think
I've
seen
that
retry
rate
before
it's
it's
when
I
think
both
both
bird
controller
and
vert
handler
both
want
to
update
the
node
structure
with
labels,
and
so
that
you
know
whoever
wins
the
other
has
to
back
off
and
retry
or
something
along
those
lines.
I
I
I
saw
lots
of
retrials
anyway
as
well.
A
A
E
A
Okay,
I'm
going
to
add
I'll.
Add
these
to
the
to
that,
like
catch-all
card
that
has
like
the
efficiency
of
the
control
loop
efficiency,
whatever
it
is,
that
I
remember
know
what
it's
called
that
shoes
I'll
find
it
on
that
we
have
it
in
six
scale
document
but
I'll
add
those
to
it,
but
it
has
additional
data
points.
Okay,
unfinished
work.
A
We
kind
of
saw
this
earlier.
This
was,
I
know
we
already
talked
about
this.
I
mean
yeah,
so
we
have
a
bunch
of
unfinished
work.
Yeah,
eight,
the
node
and
bmi
controller-
do
not
have
much,
but
I
mean
still
it
needs
sec
37
seconds,
eight
minutes.
I
guess
this
is
the
total
count
right
like
it's,
the
total
amount
of
time
spent,
waiting
or
or
stuck
like
it
could
be
like
it's
it's
an
accumulation.
It's
not
just
like
one
thing
right,
that's
I
guess
I
I
just
depends
how
you
put
the
metric
together.
A
A
I
wonder,
marcelo
like
after
you
delete
right
here
how
long
this
takes
to
go
down
kind
of
what
it
looks
like
over
time.
I
mean
we
can
see
kind
of.
We
see
a
slight
dip
here,
but
it's
still
kind
of
high
and
we've
deleted
the
vms.
At
this
point,
where's
the
count
where's
our
account
to
compare
right
here.
A
A
Okay,
I
was
looking
at
them
as
if
they
were
they
were
combined,
but
I
think
the
purple
one
is
the
one
like.
So
it
goes
at
a
max
of
484,
but
it's
actually
like.
We
see
good
peaks
and
it
looks
like
it
comes
down
to
pretty
close
to
what
baseline
was
so
that
actually
looks
fine,
because
these
are
like
again
we're
saying
these
are
stacked.
Then
this
is
then
they're
all
doing
that
some
of
these
aren't
as
well,
but
it
looks
like
they
eventually
get
there.
A
A
A
E
B
D
B
E
B
B
E
F
E
So
if,
but
it's
related
to
the
the
previous
one,
also
for
the
storage
operation,
okay,
so
yeah
we
have
the
another.
I
have
another
file
showing
things
when
I
increase
the
you
know
the
rate
level
actually
the
the
burst
and
pairs
per
second
okay.
So
we
can
see
here
that
I
was
mentioned
to
delete
the
vm,
so
I
have
like,
in
this
experiment
here
very
low
performance.
When
I
delete
the
vms,
it
sometimes
takes
hours
to
delete
vms
or
doesn't
delete.
So
I
need
to
force
the
deletion
of
the
vms
later.
E
You
know
by
hand-
and
this
is
directly
related
to
many
storage
operation
neighbors.
You
can
see
here
well
in
the
figure
in
the
right
part.
The
storage
operation
rehearse
it's
okay,
so
the
vmware
that
we're
creating
it's
using
ephemeral
volumes
with
empty
direct
plates,
the
empty
gear.
Maybe
different
volumes,
has
different
performance,
but
that's
what
we
have
here
and
it
has
a
lot
of
empty
gear
er
to
unmount
this
and
get
stuck
to
amount
that
and
it
doesn't
delete
the
pod
and
it
remains
forever.
E
A
F
E
A
E
Just
a
little
bit
more:
oh
yes,
okay,
so
just
just
this
is
the
configuration
that
I
change.
So
it's
I
increase
the
burst
to
100
for
this
components
here,
the
carrots
per
second
to
50.,
I
think
by
default,
was
five
and
ten.
Something
like
that
before
I
also
it's
maybe
a
question
to
raman.
This
is:
is
there
other
components
here
that
we
also
tune
or
desire?
Are
all
the
components?
And
the
other
question
is
when
I
change
that
only
the
vert
controller
reboot
so
well,
we're
started.
E
E
Yes,
if
and
without
I
didn't
so-
I
didn't
have
time
to
put
all
the
figures
here.
Also,
but
the
point
is
it
didn't
change
the
vm
creation
time?
Also
it
didn't
fix
all
the
the
rate
limit.
With
this
configuration,
it's
improved,
it's
not
five
500
milliseconds
anymore.
It's
actually
250.
If
you
go
to
the
other
figure
and
you
we
don't,
I
don't
yeah,
you
see
it's
still
like.
Oh.
A
A
A
A
A
A
A
E
Yeah
this
is
this
one
yeah,
let
it
go
very
quick
from
that.
So
this
was
the
experience
to
try
to
run
500
gm.
You
know
you
know
and
actually
was
hard
to
do-
that
I
increased
this
timeouts
for
the
chemo
that
we
had
before
and
and
then
many
other.
You
know
many
other
change
numbers
of
device
and
things,
but
in
the
end
I
integrate
500
vm
once
and
actually
the
vms.
E
It's
when
it's
get
like
close
to
the
cpu,
very,
very
close
to
80
or
90
percent
of
the
cpu
utilization
in
the
old,
and
also,
I
would
say,
not
also
eight
or
nine
percent
of
memorization
and
all
things
get
very
nasty.
So
the
operations
start
to
kill
you,
even
though
it
has
one
giga
of
memory.
You
know
you
know:
version
systems
start
to
kill
some,
some.
No
some
containers
and
the
via.
So
the
the
the
operation
system
also
is
very
slow.
E
It's
it's.
I
saw
I
log
into
the
node
and
even
though
it
drops
the
cpu
utilization,
for
example,
250
something,
but
it's
everything
is
very
slow.
I
see
a
lot
of
I
animation
interrupts.
You
know
calls
in
the
kernel
and
and
some
containers
being
killed,
but
I'm
not
I'm
not
sure
exactly
what's
happening,
but
when
it's
the
it's
saturated,
the
node,
it's
get
like
unstable.
That's
the
what
I'm
saying
I
also
test
with
three
different
runtimes:
the
docker
container
g
and
cryo
and
container
d
had
better
performance.
E
E
A
This
this
creation
time
marcelo
so
we've
seen
here
so
looks
like
fifty
hundred
two
hundred
fifty
hundred
two
hundred
and
then
two
hundred
all
the
way
to
four
hundred.
It's
almost
the
same.
It's
like
like
we
hit
a
threshold
here
and
then
we'd
kind
of
we
leveled
off,
which
is
that
which
is
kind
of
interesting.
E
B
A
B
B
B
A
A
We
can
whatever
that's.
We
can
call
that
so
like.
I
think
this
would
be
another
one.
I
I
kind
of
I'll
add
this.
I
think
if
I
don't
have
it,
I'm
going
to
add
it
to
our
profile
list
marcelo
as
a
thing
I
mean
since
you're
kind
of
doing
this
all
right.
This
would
be
cool
when
you
hit
like
right
here
I
mean
this
seems
like
it's
going
to
be
exponential.
I'm
guessing
like
we're
going
to
be
up
here.
A
A
A
Yeah
I
mean
you
actually
see
like
this.
This
peak
is
not
much
of
a
difference
which
is
interesting
and
then
we
kind
of
basically
double
yeah.
I
mean
I
I'm
just
this
unfinished.
Work
is
really
just
an
interesting
one.
Maybe
I'll
attach
these
unfinished
work
ones
to
the
to
that
profiling.
I
think
getting
this
information,
that's
gonna
be
really
helpful.
It
might
give
us
some
good
stuff.
E
Yeah,
just
every
quick
update
for
the
continuous
performance
jobs.
There-
evaluation
jobs
there.
It
was
not
so
the
job
there
was
not
being
collected,
the
metrics
I'm
working
on
that
with
frederico
we
are,
we
tried
like
we
are
debugging
that
to
see
many
configuration
what's
happening,
that
the
metrics
were
not
being
collected
so
because
what
we
have
is
it's
like
a
global
prometheus,
that's
running
the
cluster,
and
then
there
is
another
cluster
that
is
created
for
the
for
the
the
job
to
run.
E
You
know
the
tasks
and
that
also
creates
a
promises,
and
this
global
parameters
needs
to
collect
the
metrics
of
this
local
prometeus
and
it
was
not
being
collected
so
so
so
we
don't
have
results
for
that,
and
the
new
graffana
dashboard
is
there.
I
think
maybe
we
will
have
it
married
soon.
So
let's
hope
for
that
yeah.