►
Description
Topics covered
-----------------------------
1:14 - Introduction (Leo & Parithosh)
3:12 - Metrics Standardization
04:04 - Prometheus
05:10 - Workflow
06:10 - Ethereum Consensus layer clients
06:38 - Why is it important?
08:03 - Standardized beacon metrics
09:09 - example
09:45 - Why is this important in the context of Altair?
11:02 - How to go about standardization/Interop?
12:40 - Resource Analysis of Consensus Clients
14:27 - Study
14:45 - Resource discussion
15:10 - Memory usage
16:01 - CPU usage
17:13 - Disk usage
19:02 - Fossil records of non-finality
20:48 - Non-finality, no DB pruning
22:50 - Clients today
23:42 - DIY Consensus Client Resources Monitoring
25:20 - Q&A
25:38 - What could a solo validator do with this data?
27:32 - What’s the current state of the unified standard of API.
30:03 - Has there been a similar conversation on standardization on Eth1 side?
32:00 - A proposal - All the basic set of metrics could be documented, which would help clients' diversity.
35:46 - Is there any public dashboard for metrics available?
37:17 - Collaborating with validator - are there any data endpoints for stakers to share this information?
38:59 - Is it possible to have a pseudo app instead of installing Prometheus & Grafana?
40:45 - What was the trigger to start looking into standardization of metrics for beacon chain developers?
43:28 - How often do you find the client-specific metrics that need to be added to the set of standards metrics?
45:11 - Who fixes the fire in Ethereum?
48:20 - How was the experience with Prater testnet debugging of the client. Did the metrics standardization help?
49:25 - What would be the best place to follow metrics analysis progress?
50:43 - Eth staker survey
53:37 - Thoughts on Grandine.
Resources
-------------------
Slides - https://docs.google.com/presentation/d/1Kew49FbisRZD_t0nJLpJMhdNtyoq7N4vHLB3_KMvsfg/edit#slide=id.gf276beab0a_0_0
A webpage on BSC metrics standardization process - http://migalabs.es/eth2-client-metrics/
Beacon chain proposal - https://www.youtube.com/playlist?list=PL4cwHXAawZxoEw29YmqJtNoFaENUUAREn
PEEPanEIP playlist - https://www.youtube.com/playlist?list=PL4cwHXAawZxqu0PKKyMzG_3BJV_xZTi1F
Follow at Twitter
-----------------------------------------------
Parithosh Jayanthi @parithosh_j | Leonardo Gomez @miga_labs | Colfax Selby @colfaxselby | Pooja Ranjan @poojaranjan19
Check out upcoming EIPs in Peep an EIP series at https://github.com/ethereum-cat-herders/PM/projects/2
Contact Ethereum Cat Herders
---------------------------------------------------
Discord: https://discord.io/ethereumcatherders
Twitter: https://twitter.com/EthCatHerders
Medium: https://medium.com/ethereum-cat-herders
GitHub: https://github.com/ethereum-cat-herde...
Email: support@ethereumcatherders.com
Website: https://www.ethereumcatherders.com/
A
Welcome
to
another
episode
of
penny,
an
educational
video
series
where
we
get
an
introduction
to
the
vegan
improvement
proposals,
upcoming
features,
upgrades
and
basic
competitive
blockchain,
I'm
your
host
pooja
randhan
and
I'm
joined
by
colfax
and
noland
from
the
eatstaker
community.
In
this
episode
on
the
beacon
chain
and
benchmarking,
we
will
have
an
overview
of
the
standardized,
metrics,
interop
ability
and
the
importance
of
these
metrics
worldwide
upgrade,
along
with
the
resource
analysis
of
consensus
client.
A
To
talk
more
on
this,
I
welcome
our
guest
paritos,
a
devops
engineer
at
ethereum
foundation,
leonardo
gomez,
a
researcher
at
barcelona
supercomputing
center.
Thank
you
for
joining
us.
As
I
understand
we
have
a
presentation
coming
up
and
I'm
looking
forward
to
follow
that.
But
before
that,
could
we
have
just
an
introduction
and
a
little
bit
about
how
you
joined
the
ethereum
community.
B
B
I
usually
do
research
in
high
performance,
computing,
supercomputers
in
the
subjects
of
fault,
tolerance
and
reliability
and
scalability
and
since
2018,
more
or
less
so
about
three
years
ago.
I
started
to
be
interested
on
on
this
credibility
issues
in
blockchain
technology
and
at
some
point
I
decided
to
join
a
little
bit
this
effort
on
bringing
blockchain
technology
to
the
next
level,
and
so
I
started
to
be
involved
with
the
term
foundation
and
to
do
research
supporting
the
development
of
the
ethereum
2.0
protocol
and
so
yeah.
B
C
Hi,
I'm
paris
and
I
pretty
much
started
with
the
tvm
around
around.
When
I
was
doing
my
masters,
I
was
looking
for
an
idea
for
my
master's
thesis
and
decided
to
deep
dive
a
bit
into
ethereum
there
and
subsequently,
when
I
was
looking
for
my
first
job,
I
decided
to
work
at
a
blockchain-based
startup
at
some
point.
Last
year
I
remember
seeing
a
tweet
from
justin
from
the
ethereum
foundation
that
they
were
hiring
a
lot
of.
You
know
people
for
the
eth2
team
to
help
support
the
efforts
there.
A
C
So
starting
off
hi,
I'm
I'll
be
talking
about
metric
standardization.
Today
before
we
begin
what
are
metrics
in
itself
well,
metrics
are
quantitative
assessments
that
you
can
use
to
track
how
something
is
performing
when
you're
able
to
track
them.
You
can
use
this
to
influence
your
future
decisions
to
track
changes
to
see
if
any
changes
you
may
actually
work.
C
A
very
common
form
using
metrics
are
time
series
metrics,
so
data
points
that
are
collected
over
time
intervals.
A
simple
example
for
webmatics
can
be
useful.
You
can
see
a
graph
of
internet
traffic
at
my
household,
so
if
I
have
to
upgrade
my
internet,
I
would
have.
I
would
be
able
to
see
this
graph
and
see
if
I'm
hitting
the
bottleneck
and
know
that
I
need
to
upgrade
or
not.
C
Prometheus
is
an
open
source
monitoring
tool.
It's
one
of
the
most
famous
in
in
the
monitoring
space.
At
least
it's
part
of
the
cncf,
the
cloud
native
computer
foundation,
and
the
reason
this
is
important
is
that
it
has
a
lot
of
integrations
with
other
cloud
native
tools.
It
makes
it
extremely
interoperable
and
easy
to
use
and
plug
in
some
of
them
that
you
might
have
heard
of
would
be
like
kubernetes
an
orchestration
engine.
C
It
also
has
a
query
language
built.
On
top
of
it,
I
mean
help
you
make
sense
of
this
data,
it's
superb
it's
relatively
efficient,
and
it
gives
you
a
lot
of
options
for
scaling
and
configuring
it
in
different
manners.
Grafana
is
a
tool
we
use
for
visualization
and
prometheus.
Has
great
integration
with
grafana,
so
you're
able
to
collect
your
data
and
you're
able
to
visualize
it
and
alert
manager
is
another
tool
built
by
the
team
and
prometheus
that
can
be
used
for
getting
alerts
from
from
this
database.
C
C
Grafana
can
then
be
used
to
visualize
this
information,
in
my
case
that
88
of
the
disk
is
full.
You
could
then
use
alert
manager
to
define
alerts.
I've
just
omitted
the
expression
so
that
it
fits
in
the
image,
but
you
could
define
an
alert
that
fires
when
your
host
is
out
of
memory,
for
example,
this
alert
can
be
broadcasted
by
different
methods.
One
such
method
is
what
I've
shown
in
the
image
which
is
pagerduty.
You
could
also
get
notifications
by
telegram
email
sms,
whatever
you
configure
it
to
so.
C
C
C
But
there's
still
some
parts
that
are
missing.
For
example,
the
metric
names
don't
always
match
between
clients.
So
this
becomes
an
issue
when
you
want
to
create
generic
alerts
that
apply
over
all
clients,
or
if
you
have
a
dashboard,
you
wouldn't
be
able
to
use
the
exact
same
dashboard
if
you
want
to
switch
clients.
C
So
migrating
away
from
a
client
has
a
lot
of
friction
because
you
might
have
to
re-develop
your
alerting,
you
might
have
to
re-develop
your
dashboards
and
maybe
even
change
some
custom
tooling,
that
you've
built
a
standard
for
such
metrics
would
help
reduce
the
friction.
So
you
could
have
shared
dashboards.
Just
build
ones.
Use
always
could
have
shared
other
rules.
The
community
can
define
what
they
think
is
a
good
alerts
and
share
them,
and
you
don't
have
to
worry
about
which
client
you
have,
but
rather
just
be
able
to
use
them.
C
In
order
to
achieve
this,
we
wanted
to
look
at
a
standard
set
of
metrics.
We
just
began
with
something
called
a
minimum
metric
set.
The
way
we
chose
this
minimal
metric
set
is
leo's.
Team
created
painstakingly
created
a
really
big
table
where
they
lock
the
current
status
of
everything.
So
we
looked
at
the
metrics
that
require
the
least
amount
of
effort
from
all
the
client
teams,
but
was
still
important
and
chose
those
as
the
minimum
metric
set.
So
we
want
to
first
achieve
interoperability
between
this.
C
This
set
of
eight
metrics
over
here
and
the
hope
is
to
grow
the
set
of
metrics
to.
I
think
the
second
version
has
another
30
or
40
different
metric
labels
that
we
want
to
standardize
on,
and
we
want
to
have
a
really
robust
set
for
most
general
use
cases
to
be
covered
again.
I
will
reiterate:
this
doesn't
mean
that
everyone
has
to
follow
the
exact
same
rules.
This
isn't
like
a
spec,
but
rather
like
a
guide,
so
you
could
have
something
that's
easier
to
begin
with.
C
Just
to
give
you
an
example,
I
took
a
bunch
of
the
piermont
notes
that
we
run,
and
this
is
the
latest
version
on
all
of
them
and,
as
you
can
see,
using
the
same
label
there's
not
always
all
the
client
metrics
showing
up.
It's
also
not
different.
There
isn't
one
client
missing
in
all
of
them.
It's
a
complete
mixed
bag
right
now.
We're
hoping
that
over
the
next
couple
of
releases
that
at
least
the
minimum.
C
C
Well,
if
you
have
metric
standards,
it
makes
it
a
bit
easier
to
track
how
the
network
is
evolving
and
how
the
health
of
the
network
is
to
begin
with.
You
could
have
a
lot
more
network
level,
alerting
that
you
can
implement.
So
you
know
it
isn't
necessarily
just
one
client
that
might
be
down,
but
rather
know
be
able
to
scrape
a
wide
variety
of
clients
and
be
able
to
decide
that
the
network
itself
is
healthy
or
unhealthy.
C
Client
like
consensus
layer,
client
diversity
is
still
not
as
robust
as
we
like
to
be.
So
a
standard
in
this
sense
might
make
the
change
easier.
It
might
entice
at
least
some
solo
stakers
or
loudspeakers
to
move
away
from
what
the
setups
they
have
right
now
and
also
as
a
consensus
layer
becomes
more
important,
because
we
need
to
ensure
that
we
can
have
an
easy
setup
and
one
of
the
things
that
makes
setups
difficult
is
for
people
to
have
extremely
specified
pre-knowledge
right.
C
It
would
be
much
nicer
if
you
can
just
find
a
solo,
staker
and
say
here
there's
a
shared
dashboard.
This
will
work
irrespective
of
what
client
you
use,
and
that
being
said,
I'd
also
like
to
take
this
opportunity
to
just
tell
you
how
standardization
could
work.
Maybe
someone
amongst
you
has
an
idea
for
some
other
area
that
needs
to
be
standardized.
C
B
All
right,
thank
you
very
much
party.
Yes,
so
I'm
gonna
talk
a
little
bit
about
benchmarking
and
the
resource
analysis
of
the
consensus
clients
that
we
did
last
year.
I
would
like
to
acknowledge
my
students,
mikhail
and
luca,
for
this
work.
They
did
most
of
the
big
effort
here,
so
I
want
to
acknowledge
this.
B
B
Now,
when
we
look
at
cpu
usage,
we
see
that
again.
There
are
two
clients
that
do
use
a
lot
of
cpu,
tekku
and
lighthouse,
or
at
least
they
did
at
that
point
in
time.
Prisma
lost
stars
usually
stay
around
50
of
cpu
usage
and
again
nimbus
show
rather
low
resource
consumption
in
in
this
case,
most
of
the
time
under
50
percent
of
cpu
usage.
B
One
thing
that
we
can
see
again
here
is
that
around
hour
23,
so
here
the
x
axis
is
the
is
the
time
and
the
y-axis
is
the
cpu
usage
when
you
see
close
to
hour
24.
So
we
run
this
in
for
about
48
hours,
all
the
clients,
and
by
the
way
I
just
forgot
to
mention
that
they
were
all
running
in
the
same
node
in
the
same
specs.
So
there
is
no
difference
on
the
on
the
hardware
for
for
for
them.
B
B
That
is
exactly
when
the
cpu
usage
drops
to
zero
and
the
reason
is
basically,
the
client
could
not
continue
syncing
because
he
has
used
all
the
available
disk
in
the
in
the
machine.
So
it's
completely
normal
that
we
see
the
disk
usage
flattening
and
the
cpu
usage
drops
to
zero,
because
there
is
nothing
else
that
clan
could
do.
B
But
when
we
look
at
this
usage
was
not
only
lighthouse,
we
behave
strangely.
Actually,
if
you
look
at
tekku
at
our
maybe
five,
there
is
also
kind
of
a
spike.
There
is
the
this
orange
line.
There
is
a
a
a
a
a
kind
of
outlier
here
and
if
you
look
at
for
a
nimbus
also,
there
is
a
clearly
sharpen
increase
sharp
increase
in
in
disk
usage
after
hour.
Six
and
again
for
load.
B
Starter
is
also
a
significant
difference
at
some
point
now
you
might
think
that
they,
they
are
not
synced
like
his
first
lighthouse,
then
tekku,
then
lots
start
then
nimbus,
but
actually
this
is
because
we
were
running
the
the
clients,
the
different
clients
at
different
times.
Now,
if
we
sync
them
all
with
respect
to
the
slot.
B
So
if
you
go
to
the
next
slide,
when
we
sync
this
disk
usage
by
a
slot,
instead
of
time
of
one
o'clock
time,
you
will
see
that
the
change
in
behavior
happens
to
all
of
the
clients
at
around
slot,
70
000,
more
or
less,
and
you
see
that
at
this
point,
all
of
them
lighthouse,
tekku,
nimbus,
lots,
etc.
They
have
a
completely
different
behavior
in
terms
of
disk
usage.
Okay.
B
So,
in
addition
to
that,
when
you
look
at
the
chain
synchronization
between
slots,
70,
000
and
100
000,
there
is
also
quite
a
increase
in
a
speed,
a
change
in
the
speed
of
synchronization
for
all
the
clients
which
actually
matches
this
segment
here,
where
these
queues
exchange
for
all
the
clients.
Now.
B
What
was
happening,
then,
is
that
we
had
a
non-finality
period
on
the
media
test
net,
and
this
was
because
of
some
bug
issues
that
one
of
the
clients
had
with
respect
to
timings
and
I'm
not
going
to
get
into
the
details.
But
again,
the
important
thing
here
is
that
we
we
run
into
a
non-finality
period.
But
what
is
remarkable
to
me
from
this
experiment
is
that
we
actually
could
observe
that
just
by
looking
at
the
resource,
consumption
of
the
clients
in
your
hardware.
B
So
for
me
this
was
a
completely
you
know
unexpected
result,
because
I
was
not
thinking
that
you
could
actually
observe
the
consequences
of
you
know:
protocol
issues
in
the
blockchain
in
when
you
look
at
the
resource
consumption
of
the
grant,
but
of
course,
in
retrospective,
it
does
make
sense
that
when
you
have
a
non-financial
period,
then
you
cannot
prune.
And
if
you
go
to
the
next
slide,
you
will
see
what
I
mean
this
is.
B
These
are
the
log
events
for
one
of
the
clients
for
lighthouse,
and
what
you
see
here
for
in
the
in
the
figure
on
the
left
is
the
number
of
times
a
block
is
cured,
and
please
note
that
the
y-axis
is
a
logarithmic
scale.
Okay,
so
it's
not
a
linear
scale.
So
you
most
of
the
time
you
will
have
between
one
and
ten
blocks
cubed.
But
then
at
some
point,
when
this
is
when
the
non-finality
period
started,
you
would
have
over
200
blocks
would
be
queued.
B
At
the
same
time,
and
then
basically
zero
for
a
long
period
of
time,
okay,
so
this
is
completely
abnormal
behavior,
and
this
is
this.
This
reflects
clearly
the
non-finality
period
that
we
we
had
on
the
media
testnet
at
that
point
and
when
you
look
at
the
figure
on
the
right,
we
look
at
different
events
on
the
on
the
logs
of
the
of
the
client,
and
you
see
that
setup
events
just
stopped
happening
so,
for
example,
database
running
extra
pruning
starting
the
database
problem.
B
So
all
these
events
just
didn't
happen
at
all,
and
the
reason
why
this
problem
does
not
didn't
happen
is
because
this
is
done
after
you've
reached
finality.
So
when
you
you
don't
reach
finality,
basically
they
never
happen,
and
then
that's
why
you're
that's
when
your
this
consumption
explodes
and
basically
you're
at
some
point
your
client
kind
of
dies,
because
you
have
used
all
the
resources
available
in
the
system,
so
this
was
kind
of
an
experiment
for
us
to
see
just
curious
to
see.
You
know
how
these
different
resource
consumption
are
between
clients.
B
But
again,
what
was
surprising
for
us
was
that
we
could
actually
observe
the
consequences
of
non-finality
in
resource
consumption.
So
I
think
this
kind
of
highlights
the
importance
of
you
know
benchmarking
and
monitoring
and
having
a
common
metrics
between
clients,
because
actually
we
can
observe
much
more
things
that
what
we,
when
we
respect
now,
when
you
you
look
at
the
clients
today,
we
still
monitor
them
and
the
pictures.
B
The
figures
that
you
see
here
are
basically
figures
that
show
the
memory
usage
and
the
use
research
of
the
clients
today
running
on
the
mainnet.
So
you
see
you
know
we,
we
don't
see
all
the
weird
patterns
that
I
just
show
for
the
media
test
net,
but
we
do
see
common.
However,
some
differences
between
clients
that
is
still
important
right.
We
still
see,
for
example,
memory
consumption
is
high
for
deku
is
low
for
lingus.
B
We
see
this
usage
is
high
for
lighthouse,
see
particularly
low
for
english
again
and
you
do
see
some
patterns,
so
there
are
some
differences
that
are
still
important
between
plants,
and
so
we
we
still
monitor
this
today
and
and
if
you
want
to
do
it
yourself,
actually
you
can
so
we
have
made
these
resources
available
online.
You
can
just
go
to
this
repository
and
it's
just
a
very
simple
python
script.
B
So
I
want
to
you
know,
make
mention
explicitly
that
this
is
not
going
to
release
any
information
about
your
validator
or
you
know
the
status
of
your
validator
or
your
bitcoin
node
or
whatever
just
this
is
just
basically
monitoring
resource
consumption
of
a
unix
process.
Okay,
so
this
is
absolutely
no
information
that
is
going
to
be
released
for
your
validator.
So
if
you
want
to
try
go
ahead
and
try,
you
will
see
that
the
output
of
the
of
this
monitoring
script
is
what
you
have
seen
you
see
in
this.
B
In
this
figure,
it's
a
csv
file
that
you
can
then
plot
and
we
offer
also
a
plotter
tool
so
that
you
can
easily
plot
the
data
and
see
how
this
looks.
So.
I
encourage
all
of
you
to
who
wants
to
try
just
go
ahead
and
go
to
the
repository
and
give
it
a
try,
so
that,
if
you
are
running
nodes,
you
can,
you
know,
maybe
monitor
their
resources
consumption,
and
perhaps
you
can
even
set
some
alarms
in
case.
B
A
E
Yeah,
I
thought
of
a
few
questions
throughout
the
first
of
all.
I
just
want
to
say
awesome
presentation,
guys.
I
was
really
cool
to
see
what
you
guys
are
doing
with
the
metrics
and
then
like
the
aggregated
metrics
on
test
nets
and
whatnot.
It's
it's
really
cool.
I
thought
of
a
few
things
I
guess
I'll
start
off
with
maybe
the
simplest
one,
so
I
like
say
I'm
at
home,
solo
staker,
maybe
moderately
technical.
I
see
this
video.
E
C
So
one
of
the
nicest
things
is
alerting
for
me,
especially
as
a
homesteader.
You
might
want
to
know
when
your
disk
is
full
if
your
bandwidth
is
being
hit.
If
your
nodes
are
not
syncing
anymore
and
those
are
relatively
easy
alerts
to
set
up,
hopefully
in
the
future,
also
compatible
alerts
across
clients,
and
the
next
step
would
be
whenever
you're
thinking
of
upgrading,
for
example,
either
for
your
hardware
or
for
software
or
for
any
particular
reason,
have
a
look
at
the
dashboards
first
see
what
they
are
or
perform.
C
B
D
B
E
Cool
that's
helpful
on
the
topic
of
standardization,
of
a
metrics
api.
I
understand
you
guys
have
worked
with
the
metrics
apis
of
many
different
clients.
All
of
them.
It
seems
like,
what's
the
current
state
of
the
metrics
apis
across
clients
like
how
far
are
we
from
a
unified
standard
like?
Is
it
totally
different?
Is
there
like,
maybe
50
percent
overlap,
or
what
does
it
look
like
to
you
guys.
B
So
at
this
point,
as
party
mentioned,
we
have
a
minimal
set
of
metrics
that
we
want
to
be
standard
across
our
clients.
This
is
just
a
set
of
eight
metrics,
and
not
all
of
the
clients
have
released.
I
mean
have
adopted
these
standards,
yet
in
their
last
release,
I
thought
almost
all
of
them
are
in
the
way
of
doing
it.
B
We
just
did
a
cross
check
across
you
know
across
all
clients
this
week,
looking
at
32,
different
metrics-
and
there
is
a
lot
of
there-
is
a
lot
of
differences
between
them.
The
names
are
very
well
are
different.
There
are
several
metrics
actually
most
of
metrics.
I
think
that
are
not
supported
by
at
least
two
of
the
clients,
so
they
have
like
you
know
different
sets
of
metrics
each
one
of
the
clients,
and
there
is
no
like
a
very
clear
intersection
like
okay.
B
There
is
also
some
metrics
that
are
implemented
by
all
of
the
clients
or
some
of
the
clients,
but
with
significant
different
with
some
differences
like
they,
they
mean
the
same
thing,
but
they
are
implemented
in
a
different
way,
and
so
we
have
to
look
at
these
small
differences
and
and
to
make
it
you
know
standard
so
that
you
can
actually
set
up
a
dashboard
that
looks
at
all
different
clients.
At
the
same
time,.
E
Cool
that
makes
sense,
and
so
I
guess
that
just
immediately
brings
to
mind.
I
mean
this-
isn't
our
first
time
coordinating
or
our
collectively
and
ethereum
coordinating
clients
communicating.
We
have
like
the
existing
sort
of
eth1
execution
layer.
Clients
like
has
there
been
a
similar
conversation
on
the
historically
like
on
the
I
guess,
eth
one
side,
and
how
do
they
solve.
C
It,
I
guess,
I'm
actually
not
sure
if
they've
had
such
a
conversation
on
the
eth1
side,
yet
because
most
of
the
eth1
clients
don't
necessarily
speak
the
same
protocol
to
begin
with,
like
someone
or
one
of
the
clients
might
be
using
influx
db,
the
other
one
might
be
using
prometheus
and
so
on.
So
I
think
much
I
feel
like
they
have
a
decent
overlap
in
terms
of
metrics
themselves.
You
might
need
a
bit
more
configuration
to
get
it
there,
but
the
e3
scenario
is
actually
quite
healthy
from
there
on.
C
B
Exactly
and
I
I
want
to
yeah
emphasize
over
what
I
say-
that
the
fact
that
we
are
starting
now
so
early
in
the
development
of
the
three
and
two
is
really
a
good
thing.
You
know
it's,
it's
really
something
that
will
allow
us
to
have
something
in
the
future.
That
is
very
standardized.
If
we
had
started
this,
I
don't
know
in
one
year
or
two
years
later
in
the
future.
B
E
E
D
Not
really
a
question
but
more
likely
a
proposal
to
help
this
effort
get
along.
We
know
that,
like
writing.
Documentation
is
like
the
most
hated
job
in
development
right
so,
and
I
see
that
with
the
e2
rpc
api
there's
already
like
a
spec
out
there,
even
for
the
metrics,
you
don't
or
you
guys,
really
don't
want
to
do
a
spec,
but
having
or
something
that's
always
difficult
in
metric
land,
especially
with
prometheus,
is
to
know
what
a
metric
actually
means,
what
it
says
and
what
you
should
look
for.
D
So
it's
less
friction
to
just
follow
the
standard
than
to
come
up
with
all
metrics
and
have
to
document
and
explain
them
yourself
and
speaking
as
a
as
someone
who
is
involved
with
leiden
staking.
It's
also
very
important
for
the
for
the
big
stakers
that
we
can
diversify
clients
with
like
as
little
as
effort
as
possible
and
monitoring
is
a
huge
part
of
the
operation
because,
like
a
whole,
stacker
can
look
at
like
the
client
log
for
like
one
key
or
two
keys.
D
We
are
running
like
thousands
of
keys
and
you
can't
look
at
them
individually.
You
have
to
look
at
them
aggregated
and
you
have
to
have
all
the
automatic
alerting
for
those
to
see
if
one
of
your
nodes
or
one
of
your
validators
goes
rocky
so
yeah.
It's
super
important
to
to
have
that
on
that
side
as
well.
C
Yeah
definitely
agree
with
that.
Maybe
that's
also
a
good
idea
to
have
some
dashboards
as
in
this
in
the
spec
repo
or
whatever
we
have
as
a
spectrico
as
well.
So
people
can
already
import
it
and
start
off
from
that
and
totally
agree
with
the
alerting
as
well.
It
might
be
nice
to
have
some
alert
templates
and
maybe
bounce
those
off
along
with
spec
descriptions.
B
Yeah
internally,
we
do
have
a
spreadsheet
that
we
share
with
all
the
client
teams.
So
as
as
parie
mentioned,
we
have
select.
We
have
asked
all
the
client
teams
to
select
one
person
that
will
be
kind
of
in
charge
of
this
standardization
process
and
we
have
a
discord
group
where
we
all
exchange
information
and
we
discuss
about
what
exactly
a
metric
means,
and
so
we
have
this
spreadsheet
with
all
the
metrics
that
we
are
standardizing
with
our
full
description
of
the
metric.
And
so
I
agree
with
you.
B
A
Well,
I
think
that
partially
answered
my
next
question
that
I
was
about
to
ask.
It
was
about
resource
monitoring
that
you
mentioned
in
one
of
your
slides
that
there
are
you
know
sample
and
people
can
diy
and
have
that
done?
Am
I
correct
to
assume
that
there
is
no
public
dashboard
for
all
clients,
as
of
now
that
people
can
go
and
refer
to
just
like
excluded
or
something.
B
A
B
B
One
of
the
issues,
then,
would
be
you
know
how
to
motivate
participation
into
this
and
how
how
reliable
are
the
conclusions
you
can
draw
from
these?
Given
that
this
is,
you
know,
voluntary
participation,
but
I
think,
even
though
you
will
have
some
maybe
statistical
issues.
Looking
at
that
data,
it
is
much
better
to
have
that
that
to
have
nothing.
So
I
think,
is
it's
going
to
be
useful,
yeah,
so
yeah
we
are
looking
into
that.
C
C
You
just
have
to
do
some
initial
setup
to
configure
where
that
server
is
and
one
such
playbook
is
actually
set
up
the
entire
metrics
infrastructure
for
you.
So
it
would
be
as
simple
as
saying
ansible
playbook
a
file
that
says
where
the
server
is
with
the
ip
address
and
which
setup
metric
playbook
and
then
you
would
have
met.
You
would
have
a
grafana
instance
with
a
dashboard
and
so
on.
A
So
I
have
maybe
a
follow-up
of
what
cool
facts
asked
earlier
about
the
standardization
of
metrics.
So
I'm
curious
to
understand
the
trigger
point
when
the
beacon
chain
team
or
the
devs
who
were
involved
in
the
development
of
this
chain,
started
to
look
into
it,
because
I'm
aware
like
there
was
no
such
president
eat
one
client's
right.
So.
B
So
well,
I
I
think
well,
I
think
buddy
has
a
lot
of
experience
fighting
with
dashboards
for
the
test
nets.
In
our
case,
we
we
were
just
interested
on
analyzing
how
the
different
clients
behave
and
used
benchmarking
them
and
just
document
the
the
the
differences.
It
was
just
a
very
just
simple
research
kind
of
investigative
investigative
approach,
but
then,
when
we
started
that
work,
we
started
with
promotions
and
graffana,
and
you
know
we
run
into
all
the
issues
that
you
know.
B
I
would
share
this
document
with
with
danny
and
and
others,
and
I
think
at
that
point
I
think,
is
that
we
realized
that
yeah
there
is
it's
really
hard
to
make
sense
of
any
data,
given
that
they
are
all
reporting
different
kind
of
things,
and
so
I
think
that
was
the
moment
that
we
say.
Maybe
we
should
do
something
about
this,
and
then
I
guess
afterwards,
when
paris
started
working
with
desnets,
he
also
realized
that
that
was
very
difficult.
But
I
would
probably
talk
about
that.
C
Yeah
exactly
so,
I
have
set
up
and
manage
a
lot
of
the
validators
for
the
test
nets,
especially
if
you
want
to
test.
For
example,
the
altea
update.
We
did
a
lot
of
devnets
and
metric
dashboards
are
one
of
the
easiest
ways
to
get
this
aggregate
view
of
how
the
network
is
looking
and
I
think
the
more
test
nets
that
were
set
up,
the
more
important
it
became
and
such
approach
became
more
and
more
important.
So
I
guess
that's
one
of
the
driving
factors.
C
A
C
I
think
it's
an
ongoing
process
so
currently
we're
still
trying
to
facilitate
discussions
on
the
next
set
of
metrics
to
also
get
people
to
agree
on
what
the
metric
has
to
mean
and
so
on,
and
we
always
use
leo's
leo
and
his
team's
earlier
research
as
a
trigger
point.
We
try
and
see
okay
cool,
so
three
out
of
the
four
clients
already
agree
on
what
this
is.
C
Maybe
we
just
have
to
convince
the
fourth
one
that
they
also
should
agree
or
all
three
change,
because
the
fourth
one
had
a
better
reason,
whichever
whichever
one
it
is.
We
just
want
to
facilitate
that
discussion,
and
usually
we
use
that
as
a
starting
point
and
then
go
downwards
so
easier
to
harder
at
least.
B
G
Normally,
when
you
have
this
kind
of
system
of
monitoring
for
a
company
specifically,
you
have
like
I
mean
you
guys
mentioned
leader
duty,
for
example,
and
it
kind
of
indicates
like
that
there
are
fires.
If
you
need
to
fix
fires
who
fixes
the
fire
in
ethereum,
or
is
this
particularly
related
to
the
clients.
C
Well,
that's
relatively
notorious.
It
depends
on
where
the
fire
is
to
begin
with.
If
it's
a
client
bug,
then
the
client
needs
to
issue
an
update
and
we
have
to
convince
the
community
as
a
whole
to
update
as
well
and
that
that
includes
educating
and
getting
the
word
up
to
as
many
people
as
possible.
If
it's
a
specific
validator
or
an
operator,
if
or
a
larger,
staking
service,
that's
down,
then
the
trick
is
identifying
patterns
and
seeing
who's
down
and
trying
to
poke
them
and
say,
hey,
looks
like
you
guys
are
down.
C
Maybe
your
monitoring
stack
didn't
pick
it
up,
but
the
network's
unhealthy.
Could
you
help
us
out
here?
Another
location
where
that
might
be
interesting
is
when
we're
testing
out
hard
forks
like
the
alter
heart.
For
example,
we
noticed
in
one
of
the
devnets
that
peers
were
going
like
the
ef
nodes
were
losing
kills
so
that
led
to
you,
try
and
bring
a
couple
of
time
teams
together
collect
as
much
information
as
possible
who's
losing
cares.
Why
are
they
losing
fears?
What
cares
that
they're
losing
and
then
work
backwards
to
see?
C
C
We
actually
had
this
in
one
of
the
tap
nuts,
where
we
were
able
to
figure
out
that
there
was
a
expectation
of
a
different
implementation,
so
one
of
the
clients
was
down
scoring
another
client
more
than
what
they
should
be
because
they
both
thought
of
some
concept
differently,
and
the
trigger
point
was
that
you
were
losing
cures
and
that's
easier
to
see.
If
you
have
aggregated
metrics,
because
you
can't
make
that
assumption
from
one
node,
you
need
to
make
that
assumption
from
a
network
view.
D
C
Yeah
exactly
so,
it's
a
combination
of
alert
manager,
prometheus
and
grafana
would
achieve
the
same
thing
as
what
datadog
is
doing
or
similar
thing
is
what
datadog
is
doing.
The
major
difference
is
you
have
to
self-host,
or
you
can
self-host
from
each
year's
alert
manager
and
grafana,
and
all
the
data
belongs
to
you
versus
with
datadog.
You
have
to
probably
going
to
contact
with
them
and
figure
out
how
things
work
on
their
system.
A
C
So
the
prata
debugging
for
the
sync
aggregate
stuff
was
nothing
to
do
with
the
metric
standards,
at
least
mainly
because
that's
still
extremely
new
stuff
and
I
think
proto
wrote
really
cool
tools
that
can
generate
those
fancy
graphs
that
are
really
detailed.
Maybe
in
the
future.
We
also
have
metrics
for
that,
so
that
everyone
can
use
them,
but
not
right
now,
at
least.
A
Yeah,
I
have
one
last
question
but
before
that,
if
anyone
has
any
other
question
to
go
ahead,
okay,
so
my
last
question
is
about
the
resources
like
what
would
be
the
best
place
to
follow
a
metrics
in
benchmarking.
I
remember
you
mentioned
about
the
github
repository
where
metrics,
whether
is
it
still
existing
or
has
it
changed
or
any
other
place
where
people
can
find.
C
B
A
F
Create
a
survey:
can
you
guys
see
that
so
we
are
pretty
close
to
releasing
this?
I
think
there's
just
a
few
things
with
the
website
that
we
just
want
to
make
it
a
little
bit
prettier
looking
and
nicer,
but
basically
we're
just
trying
to
get
data
on
people
that
are
already
running
nodes,
people
that
are
using
staking
services
and
people
that
maybe
are
interested
in
staking,
but
haven't
made
the
leap
yet
so
you
can
see.
F
We
start
like
getting
some
background
information
figuring
out
how
experienced
they
are
in
the
ethereum
world
and
we
ask
them
if
they're
a
solo
validator
if
they're
using
staking
service
or
they're
not
staking
at
all,
depending
on
their
answer
there
they
go
to
different
sets
of
questions
and
we're
just
looking
to
collect
some
data
figure
out
what
pain
points
are
figure
out.
You
know
what
software
people
are
using,
what
hardware
people
are
using
and
then
asking
for
feedback
on
that.
F
A
F
F
I
think
they
just
have
some
issues
with
investors
and
they
got
a
grant,
but
they
can't
make
it
open
source
and
all
that
so
until
it's
open
source.
I
can't
really
recommend
anybody
uses
it,
but
I'm
looking
forwards
to
them
sorting
out
their
issues
and
getting
an
open
source
and
then
yeah
I
mean
obviously
having
more
clients
is
awesome.
B
Yeah,
I
think
one
of
the
differences
I
heard
from
from
them
is
that
they
they
did
apparently
again
is
closer.
So
we
don't
know,
but
apparently
they
did
a
good
job
at
paralyzing,
some
parts
of
the
attestation
verification
process,
and
so
that
really
kind
of
speed
up
the
what
the
client
in
general
and
so
yeah
that
looks
very
interesting,
and
if
we
can,
we
can
look
at
this
at
some
point
in
the
future.
That
would
be
great.
C
A
A
I
also
think
it
is
important
to
share,
what's
going
on
behind
the
scene,
to
let
community
know
how
hard
developers
researchers
are
putting
into,
and
this
will
not
only
attract
brilliant
mind,
but
I
assume
that
it's
going
to
make
ethereum
the
most
transparent
and
decentralized
blockchain.
We
appreciate
all
the
hard
work
put
in
by
devops
team,
the
research
team
and
the
implementers
team
once
again,
congratulations
leo
for
the
acceptance
of
the
paper
on
e2
network
crawler
at
ieee,
estonia
conferences,
best
research
for
the
presentation.
I
hope
the
presentation
is
tomorrow
right,
yeah.
B
A
Joining
us
and
making
this
episode
more
informative
and
a
huge
thank
you,
our
youtube
viewers
for
watching
this
episode.
If
you
like,
this
episode,
hit
the
like
button
on
your
screen
and
share
it
with
community
members,
we
want
this
information
to
go
out
and
reach
as
many
as
community
members
of
ethereum.
They
know
they
participate,
they
interact
and
that's
what
our
goal
is
subscribe
to.
Ethereum
category
youtube
channel
for
meeting
and
educational
resources
we'll
be
back
with
another
interesting
topic
for
you
till
then
keep
watching
keep
sharing
your
love
with
the
ethereum
catalyst.