►
Description
Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on Swarm, IPFS and more.
https://archive.devcon.org/archive/watch/6/lodestar-metrics-driven-development/
Lodestar is the newest production-ready consensus client, written entirely in TypeScript.
How did we do it? How did we go from buggy, slow, and scared to agile, confident chads?
Metrics, metrics, metrics!
We will introduce the concept of metrics-driven development in the context of blockchain node development and give fun examples where metrics saved our hide, time and time again.
Speaker(s): Cayman Nava
Skill level: Intermediate
Track: Layer 1 Protocol
Keywords: metrics,consensus,software practices
Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/
Devcon is the Ethereum conference for developers, researchers, thinkers, and makers.
Devcon 6 was held in Bogotá, Colombia on Oct 11 - 14, 2022.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/
A
A
Really
just
our
story
and
how
we've
developed
lodestar
over
the
past
three
years
so
about
us.
My
name
is
Cayman
I
work
at
chainsafe.
It
is
a
great
company,
we're
hiring.
We
basically
build
blockchains.
We
have
a
whole
bunch
of
layer.
One
projects
lodestar
is
one
of
them,
but
if
you
are
interested
in
getting
into
core
development,
we're
a
really
great
place,
great
culture.
So
lodestar
is
it's
one
of
these
projects
at
chain
safe.
We
we're
we're,
basically
an
eth2
or
ethereum
consensus
ecosystem
written
entirely
in
typescript.
A
We
really
take
the
open
source
ethos
to
heart.
We
have
all
of
our
meetings,
they're
public
everything
we
do
is
public
and
we
really
encourage
anyone
who
wants
to
contribute.
We
we
try
to
help
people
out
and
get
get
people
involved.
A
lot
of
our
team
members
actually
started
started
their
work
on
the
project
through
their
open
source
contributions,
and
it
was
kind
of
like
a
open
source
to
contract
to
hire
kind
of
a
situation.
So
we
really
we
really
like
that.
A
That
kind
of
ethos,
but
we're
talking
about
today,
is
specifically
our
consensus,
client
and
how
we
got
from
like
prototyping
typical
kind
of
code
to
something
that's
like
production,
ready,
something
that
you
can
really
rely
on,
something
that
doesn't
just
blow
up
all
the
time.
So
before
I
kind
of
step
into
our
story,
I
just
want
to
kind
of
do
a
very
high
level
of
what
metrics
are.
This
is
I'm
not
going
to
go
too
deep.
A
Then
you
use
grafana
to
visualize
these
metrics.
So
a
picture
is
like
a
thousand
words,
so
it's
basically
the
difference
between
like
looking
at
a
bunch
of
logs
and
then
looking
at
like
pretty
graphs
that
can
show
you
give
you
a
little
bit
more
insight.
Give
you
something
at
a
higher
level
than
just
squinting.
A
So
our
story.
Basically,
we
we
started
with
really
bad,
with
not
very
much
infrastructure,
not
very
much
a
discipline
and
how
we
tackled
metrics
or
tackled
like
the
data
that
we
we
track,
and
eventually
we
kind
of
like
over
time
and
like
through
a
lot
of
pain
and
and
suffering.
We
kind
of
realized
that
metrics
are
the
way
to
go.
A
Metrics
give
you
insights
that
you'll
never
get
just
by
looking
at
logs,
so
really
metrics
aggregate
a
bunch
of
information
and
give
you
the
tools
you
need
to
make
better
decisions
about
how
you
release
your
product,
what
things
to
focus
on
like
what
are
priorities
and
they're,
also
just
really
pretty
so
they're
cool.
So
this
is
a
picture
from
the
early
days
of
lodestar
from
the
interop
walk-in
in
Muskoka
in
2019.
Here's
our
message:
yeah
we're
stuck
syncing
and
some
P2P
issues.
A
We
didn't
really
know
what
what
was
going
on
like
why
things
were
breaking.
We
were
looking
at
logs,
actually
on
the
other
side
of
the
screen
here
you
you
know,
we
look
happy,
but
it's
like.
Oh
things
are
breaking
here's
some
some
funny
issues
that
we
had.
You
can
see
priority
low,
adding
Prometheus
monitoring.
We
didn't
even
know
we
were
doing
yeah
out
of
memory
om
out
of
memory.
It
was
just
like
plaguing
a
lot
of
our
development
throughout
2019
into
2020.
A
Lodestar
was
like
very
Proto,
very
much
like
a
prototype
and
we
ran
into
the
issues
time
and
time
again
like
we
would
have.
The
same
thing
happen
multiple
times
and
we
weren't
like
like
regressions
that
were
happening
and
we
weren't
really
like
learning
our
lesson
kind
of
the
turning
point
was
you
know,
starting
to
take
some
time
to
to
Really
which
takes
the
time
to
like
build
out
these
metrics
and
build
out
these
dashboards.
A
A
Are
you
know?
Where
are
we
spending
time
in
the
in
in
our
software?
Where
are
we?
Where
is
a
bunch
of
memory
being
spent?
We
really
kind
of
uncovering
all
of
the
complexity
of
this
running
software
I
mean
blockchains
are
really
complicated.
Software
there's
a
whole
bunch
going
on
and
unless
you're
really
tracking
tracking
key
pieces,
you're
you're
kind
of
going
blind
and
now
we're
finally
kind
of
at
a
point
where
we
for
any
new
feature
that
we're
adding
we
ask
for
metrics.
A
A
So
if
you
are
adding
a
case
of
a
retry
mechanism,
you're
adding
some
new
feature,
you
need
to
make
sure
that
it's
actually
working
and
the
only
way
that
you
can
or
maybe
not
the
only
way,
but
a
really
the
really
great
way
of
doing
that
is
being
able
to
show
that
visually
being
able
to
show
okay.
Our
cache
is
now
bounded.
Let's
see,
let's
see
that
in
the
metrics,
let's
see
how
it's
How,
that
cache
is
growing
over
time
or
let's
see
how
many
times
the
retry
mechanism
is
actually
being
activated.
A
Those
are
the
sorts
of
things
that
you
really
only
look
confined
by
through,
through
these
graphical
methods,
through
these
metrics
you're,
going
to
have
a
really
hard
time
looking
at
the
logs.
For
that
the
other
great
the
other
thing
about
metrics
driven
development
is
that
any
deployed
software
that
you
use?
You
need
to
be
monitoring
it
very
carefully
for
regressions,
so
look
here:
this
is
the
process
Heap
increasing
over
time.
A
This
is
a
comment
from
Lion
for
the
last
two
days.
The
leak
is
clearly
visible:
an
impact
of
75
megabytes
a
day
yeah.
So
that's
going
to
be
a
problem
if
your
blockchain
is,
if
it's
just
increasing
at
75
megabytes
a
day
continuously
you're
going
to
run
out
of
memory-
and
these
are
this-
we
were
able
to
catch
this
this
issue
before
we
actually
before
we
actually
cut
a
release,
so
it
makes
your
release
process
a
lot
at
least
release
process
is
a
lot
better.
Here's
another
here's
another
example
of
a
regression.
A
We
deployed
deployed
a
version
of
our
software
and
we
saw
the
cache
size
grow.
It's
like
oh,
is
that
is
that
is
that
a
bug
is
that
a
is?
Is
it
okay?
Well,
we
only
actually
caught
it
because
we're
tracking
these
things,
you
probably
aren't
even
going
to
have
a
log,
that's
going
to
show
you
your
cash
sizes.
So
unless
you,
you
actually
measure
this
you're
going
to
be
blind.
A
The
other
great
thing
about
the
about
dashboards
and
about
metrics
is
that
it
lets
you
correlate
different
different
pieces
of
the
code
to
find
out
where
the
problem
is
so
here
we
have
different
you
can
you
can
see
a
bunch
of
different
graphs
kind
of
all
at
the
same
time
and
see
okay,
something
is
affecting
the
event.
The
event
Loop
lag
I
can
see.
At
the
same
time,
the
active
number
of
handles
is
growing.
A
The
number
of
requests
is
growing,
so
you
can
kind
of
you
know
kind
of
solve
the
mystery
in
a
in
a
in
a
way
that
you're
not
going
to
be
able
to
do
in
the
logs.
Another
great
strategy
is
is
using
different
versions
if
you're
running
software
so
running,
like
a
staging
version
of
your
software
against
a
production
version
or
against
several
recent
versions
and
being
able
to
overlay
data
on
top
of
on
top
of
one
another
and
being
able
to
see
okay.
A
A
These
are
very,
very
practical
kind
of
tips.
Don't
abuse
the
histograms,
so
I
said:
there's
three
different
types
of
metrics.
You
just
wouldn't
doubt
you
use
the
simplest
tool
possible.
Histograms
are
really
only
used
if
you're
wanting
to
see
the
kind
of
the
distribution
of
of
of
something.
So
like
maybe
like
looking
at
like
request
timing,
maybe
you'll
want
to
see
like
all
right.
Certain
requests
are
happening
very
very
quickly,
but
then
others
are
happening
kind
of
longer.
A
So
the
classic
mistake
with
histograms
is
is
using
a
label,
that's
unbounded.
So
if
you
had
a
metric
that
was
tracking
something
per
peer
ID
on
the
network.
Well,
you
might
have
thousands
of
peers
with
and
if
you,
each
new,
each
new
peer
is
going
to
be
using
a
new
peer,
ID,
it's
going
to
blow
the
metrics
up
and
it's
going
to
affect
the
performance
of
Prometheus
and
you're
not
going
to
be
able
to
run
the
queries
that
you
want
to
run.
A
Oh,
so,
how
do
you
know
which
metrics
to
add
so
I
showed
some
of
these
pretty
graphs
and
everything,
but
like
you're
in
your
in
your
software?
How
do
you
know
what
you
want?
What
do
you
want
to
track?
So
really,
what
you
want
to
do
is
think
about
questions
that
you
couldn't
ask
otherwise.
So
a
really
good
example
is
the
size
of
your
internal
caches.
A
Another
thing
you
can
think
about
is
like
if
this
feature
or
if
this
part
of
the
code
would
degrade
or
explode
in
the
in
the
kind
of
bad
case
that
that's
a
that's
a
prime
candidate
for
for
adding
metrics
around
it
and
then
really
just
like
keep
asking
questions,
and
then
it's
like.
If
you
have
the
data
to
be
able
to
answer
the
question,
you're
good,
but
otherwise
keep
adding
metrics
until
you
can,
until
you
can
answer
those
questions,
so
some
examples
that
we
we've
kind
of
come
across.
A
How
often
are
our
streams
being
reset?
How
many
peers
do
we
have?
These
are
kinds
of
questions
that
that
the
metrics
can
help
and
so
I'm
going
to
show
you
just
a
live
demo
of
or
a
live
example
of
of
our
dashboards,
and
we
can
kind
of
run
through
some
things,
and
if
anyone
has
any
questions
about
it,
while
I'm
going
through
it,
we
can.
We
can
do
that
too.
Okay,
so.
B
A
Is
this
is
a
our
dashboard
for
our
Fleet
of
of
Beacon
nodes
and
validators?
This
is
kind
of
the
high
level
dashboard
right
now
we're
looking
at
an
unstable
node.
We
can
go
to
like
stable
a
stable
version
of
lodestar,
so
this
is
running
version
1.1
or
on
the
girly
Network
we're
synced.
It
looks
like
it
got
restarted
about
a
day
ago.
A
Some
interesting
things.
I
guess
you
know
peer
peer
count
is
looking
good
all
right,
you
know,
there's
nothing
everything
looks
stable
there.
We
track
different
types
of
peers,
inbound
outbound,
and
for
us
we
really
care
about
the
number
of
peers
that
we're
going
to
be
receiving
messages
from
kind
of
continuously.
Those
are
our
gossip
peers,
so
the
number
of
mesh
piers
on
our
core
topics.
A
A
So
one
thing,
one,
one
area
that
we
really
struggled
with
and
where
we
added
a
lot
of
metrics
is
in
our
gossip
sub-implementation.
We
found
through
adding
a
few
metrics
that
we
weren't
getting
blocks
on
time
and
it
turned
out
that
we,
we
didn't,
have
enough
peers
who
were
sending
us
these
who
are
setting
us
blocks,
and
then
we
started
asking
questions
all
right.
Well,
why
are
they
not
sending
us
blocks?
It
turned
out.
It
was
because
our
peer
scoring
was
too
low
all
right.
Well,
why?
Why
was
the
score
too
low?
A
A
A
What
we
were
seeing
before
is
you
know:
you'd
see
a
graph
where
the
values
are
like
going
down
and
down
and
down
over
time
or,
if
they're,
like
very
negative
numbers,
another
really
useful
metric
that
we
were
that
we
check
a
lot.
Is
things
related
to
the
VM,
so
memory,
CPU
and
disk
storage
so
looks
like
our
memory
is
stable?
That's
great
GC
is
only
taking
roughly
seven
and
a
half
percent.
This
is
things
are
looking
good.
A
C
Hello
and
thank
you
for
the
talk
so
I
see
that
you
have
a
ton
of
metrics
and
I
believe
that
most
of
them
are
useful.
But
do
you
have
some
kind
of
an
alerting
system
for
you
to
be
able
to
know
when
a
metric
has
gone
wrong
because
I
think
it's
just
unfeasible
to
go
through
each
of
them
like
daily
or
something
like
that?
You
I
think
you
need
some
automated
way
to
to
alert
you
that
something
is
going
wrong.
A
Yeah,
absolutely
so
that
was
something
I
didn't
touch
on,
but
it
definitely
feeds
in
so
we
we
use
pagerduty,
you
can.
You
can
basically
set
up
thresholds
for
certain
metrics
that
trigger
either
slack
notifications
or
Discord
notifications
or
in
the
case
of
a
critical
kind
of
issue.
It'll
call
you
or
text
you
yeah,
we
do
have
that
set
up.
I
would
definitely
recommend
that
if
you
are
in
running
something
in
production.
A
D
You
again,
there
are
lots
of
metrics.
Do
you
did
you
guys
develop
over
time
a
way
to,
or
you
know,
kind
of
a
mental
model
or
framework
to
decide
how
to
where
to
put
them?
Whenever
you
have
new,
you
know
graph
or
metric
instead
of
or
do
you
just
like
decide
to
put
it
somewhere
and
then
go
back
and
like
you
know,
to
make
it
easy
to
to
read
or
to
browse
the
dashboard
I,
don't
know
if
that
makes
sense.
This.
A
A
The
other
thing
that
the
other
kind
of
issue
that
we
had
initially
is
we
would
create
a
bunch
of
metrics,
but
we
wouldn't
add
them
to
the
dashboard.
So
there
was
no
actual,
like
visual
cue
for
for
a
lot
of
it,
and
so
it
kind
of
just
went
un
unnoticed,
and
so
one
thing
we
would,
we
would
definitely
recommend
would
be
like.
If
you
add
a
metric,
also
add
it
to
the
dashboard.
At
the
same
time,.
A
It
affect
the
storage
well,
so
some
of
it
kind
of
depends
on
your
architecture
of
how
you're
deploying
this.
So
what
we
do
is
we
actually
have
one
Prometheus
and
grafana
server,
servicing
all
our
entire
fleet
of
of
Beacon
nodes.
So
you
know
we
have
a
bunch
of
different
machines
and
then
we
have
one
one
machine
for
these
right
to
to
things
so
it
it
doesn't
cut
into
the
to
the
storage
we
retain
for
a
month,
quick.
A
So
yeah
a
good
question,
so
I'd
say
like
I,
don't
know
if
it's
a
half
and
half
but
like
half
the
metrics
are
like
more
like
for
like
monitoring
the
chain,
something
like
a
like.
A
block,
Explorer
kind
of
kind
of
thing,
but
I
would
say,
like
a
lot
of
the
metrics,
are
specific
to
the
implementation.
So
and
those
are
the
ones
that
are
I,
think
probably
more
helpful.
A
A
I
know
that
other
clients
do
have
metrics
and
I
I
think,
like
the
ones
that
are
more
user
focused.
Those
are
the
ones
that
look
more
like
a
block
Explorer
and
maybe
with
some
like
validator,
like
showing
your
balance
ticking
up
over
time
and
I.
Think
those
are
those
are
public
and
I
and
there
is
an
effort
to
standardize
metrics
across
the
consensus
clients,
so
I
think
some
of
that's
happening
but
again
to
the
to
the
previous
question.
A
lot
of
the
metrics
will
always
be
implementation
specific
in.
D
A
Right
so
so,
for
any
of
these
larger
feature
pull
requests.
We
have
the
the
person
who
is
authoring
or
kind
of
owning
that
that
feature
they're
responsible
for
making
the
case
that
that
that
it's
correct
and
then
it
works
as
intended
and
so
they're
responsible
for
looking
at
the
metrics
and
building
building
that
case
and
then,
as
far
as
like
our
release
process,
we
have
a
checklist
of
basic
things
that
we
look
at
before
we
before
we
actually
release.
We
have
a
testing
period,
so
we
cut
a
cut,
a
Beta
release.
A
F
A
The
the
standardization
effort
is
to
to
build
out
some
kind
of
Baseline
of
functionality
like
you
can
have
one
one
Prometheus
and
grafana
running
and
then,
whichever
client
you
want
plugged
in
here,
you
kind
of
have
some
guarantees
like
it's
not
breaking
and
like
it's,
it's
kind
of
working,
so
yeah.
That's.