►
From YouTube: Observability pre-release: Using Prometheus to test and fix new software - GitHub Universe 2018
Description
Presented by Sneha Inguva, Software Engineer, CNS Networking (DigitalOcean).
See how the team at Digital Ocean leveraged Prometheus to properly instrument and test features within their software-defined networking pillar. This session will highlight instrumentation, key visualizations, and takeaways from their experience. You’ll also hear about areas for improvement and find out how to use these learnings for your own releases.
About GitHub Universe:
GitHub Universe is a two-day conference dedicated to the creativity and curiosity of the largest software community in the world. Sessions cover topics from team culture to open source software across industries and technologies.
For more information on GitHub Universe, check the website:
https://githubuniverse.com
A
Hi
everyone
I'm
here
from
digitalocean
before
we
begin
I,
want
to
tell
you
a
little
bit
about
myself.
My
name
is
Sneha
and
guvo
I'm
a
software
engineer
at
digitalocean
currently
on
the
network
services
team.
So
we
do
a
lot
of
work
with
Software,
Defined
Networking
and
that's
a
photo
of
my
cat
for
absolutely
no
reason,
but
you
know
github
octo
cat
cat.
A
So
if
you're
not
familiar
with
digitalocean
we're
a
cloud
hosting
company,
we
have
about
12
data
centers
located
at
all
around
the
world
and
we
have
a
multitude
of
products.
So,
for
the
longest
time
we
had
compute,
essentially
virtual
machines,
storage,
object
and
block,
and
most
recently
we
have
I
guess
it's
might
be
in
beta
kubernetes
as
a
service
and
more
coming
soon.
A
We
also
operate
at
a
fairly
large
scale.
As
a
cloud
provider,
we
have
more
than
300
micro
services
in
production
today
about
1.15
million
virtual
machines
and
in
terms
of
metrics
that
are
being
scraped
about
our
micro
services
and
virtual
machines.
We
have
about
1.8
million
metric
samples
being
scraped
every
second.
So
with
that
in
mind,
I
want
to
take
you
on
a
mystical
journey
through
a
time
of
change
at
digitalocean,
so
the
past
this
was
probably
even
before
I
joined.
The
company
services
were
primarily
run
on
virtual
machines.
A
We
would
often
provision
these
virtual
machines
with
chef
we
would
deploy
to
them
using
chef
bash
scripts.
Some
other
weird
combination
in
terms
of
monitoring.
We
primarily
had
blackbox
monitoring,
so
we
used
icinga
and
not
juice.
So
this
wasn't
great
for
a
couple
of
reasons.
First
of
all,
we
would
often
spend
more
time
writing
chef
or
editing
chef
than
writing
actual
applications,
and,
secondly,
we
didn't
actually
have
a
lot
of
introspection
into
our
services.
Luckily,
there
was
a
change
and
we
entered
a
time
of
beauty
and
happiness.
A
So
now
services
have
moved
to
largely
or
I
guess
for
a
good
chunk,
largely
running
on
content
in
containers
on
Nettie's.
We
also
built
an
abstraction
layer
on
top
of
kubernetes
known
as
docc,
and
most
services
are
deployed
to
do
CC
using
this
do
CC
CLI
tool,
and
on
top
of
that,
we've
set
up
metrics
tracing
and
centralized
logging.
So
now
we
get
a
lot
more
observability
into
our
production
micro
services.
We
can
now
actually
tell
if
a
service
is
up
or
down.
A
We
can
trace
a
request
through
a
multitude
of
services,
especially
if
there's
an
error
in
our
request,
and
we
can
also
set
up
alerts
based
on
metrics
this.
Let
us
ask
another
question:
OA,
it's
great,
that
we
have
observability
into
production
services,
but
what
about
services
that
were
just
writing
new
services,
and
so
that
brings
me
to
the
topic
of
my
talk
today:
observability
pre-release,
using
Prometheus
to
test
and
fix
new
software.
A
So
the
plan
today
is
we're
first
going
to
talk
about
observability
at
digitalocean,
I'm
gonna
dive
into
one
of
the
pillars
of
observability
metrics
and
talk
about
what
we
like
to
measure
for
metrics
I'll
then
go
into
Prometheus
and
talk
about
what
exactly
it
is
and
why
we
like
it.
Finally,
I'll
get
to
the
core
of
my
talk,
which
is
about
the
build
instrument
test.
Iterate
loop
and
I'll
help
end
with
some
examples
to
really
illustrate
kind
of
what
I'm
talking
about
so
starting
with
observability
I.
A
Don't
know
when
the
term
pillars
of
observability
came
about
I
think
it
was
like
three
ish,
maybe
two
three
years
ago,
in
a
blog
post,
but
observability
generally
refers
to
introspection
into
a
system
or
service
and
the
pillars
of
observability
refer
to
metrics
logging
and
tracing
so
metrics,
essentially
a
time
series
of
sample
data.
Here
we
have
a
time
series
of
peak
go
routines
amongst
certain
instances
of
a
micro
service
over
time.
Metrics
are
great
and
that
we
can
apply
mathematical
functions
to
analyze
data
a
little
further.
A
We
can
aggregate
trends,
we
can
also
split
up
data
by
dimension
and
we
can
configure
alerts,
especially
threshold
alerts.
The
next
pillar
of
observability
is
tracing,
so
tracing
refers
to
having
timed
operations
known
as
spans
and
propagating
that
metadata
to
a
variety
of
threads
requests
and
processes.
So
here
we
have
an
example
trace
of
request.
That's
going
through
the
serve
DHCP
before
operation.
A
We
can
kind
of
see
where
how
the
request
traverses
and
the
latency
of
each
component
of
this
operation
this
is
useful
and
that
we
can
look
into
causality,
especially
if
we
have
latency
issues.
We
can
maybe
look
into
the
causality
of
that.
Latency
issue
like
which
micro-service
is
is
contributing
to
that
latency.
We
can
also
watch
a
single
request,
Traverse
multiple
services-
and
this
is
especially
cool
and
something
that
happened
I
believe
last
year
at
our
hackathon.
A
We
can
actually
use
distributed
tracing
to
build
a
map
of
the
our
overall
production
ecosystem
and
kind
of
see
how
different
services
interact
with
each
other
and
the
last
pillar.
Observability
is
logging
route,
which
refers
to
a
record
of
events
over
time.
So,
above
you
can
see
the
logs
over
time
kind
of
a
bar
chart
of
North
T.
A
The
unfortunate
thing
with
logging
is
logs
are
often
more
difficult
to
store
long
term,
but
they
can
help
you
identify
a
needle
in
the
haystack
type
issues.
So
that's
observability,
but
the
focus
of
this
talk
is
metrics,
so
I'm
going
to
talk
about
what
metrics
we
like
to
measure.
So,
if
you've
read
the
Google
SR
e-book
site,
reliability,
engineering
at
Google,
awesome
collection
of
essays,
you've,
probably
heard
of
the
four
golden
signals
which
refer
to
key
metrics
for
a
request
based
system.
So
they
refer
to
latency
the
time
to
service,
to
request
traffic.
A
The
number
of
requests
per
second
often
broken
out
by
the
type
of
request
error,
just
our
rate
of
request
and
saturation
often
a
little
bit
more
difficult
to
understand,
but
it
refers
to
fullness
of
a
service.
There's,
of
course,
the
use
metrics
for
resources
on
this
I
believe
this
term.
This
acronym
was
coined
by
Brendan
Gregg
and
used
refers
to
utilization
time
a
resources
spent
servicing
work
saturation,
maybe
but
the
extra
work
that
a
resource
can't
service
as
well
as
error
rate
just
similar
to
before
just
the
rate
of
error
events
and
Brendan
Gregg.
A
A
So
those
are
our
metrics
that
we
care
about.
But
how
do
we
measure
them?
And
that's
where
Prometheus
comes
in
here's
the
beautiful
Prometheus
logo?
It
is
a
pull
based
metric
solution,
pull
based,
meaning
the
Prometheus
instance
actually
scrapes
metrics
from
our
targets
of
choice,
rather
than
their
targets
of
choice
or
the
services
pushing
metrics
choose
our
monitoring
solution.
So
the
push
versus
pull
debate
is
something
that
has
been
around
forever,
but
I'm
not
gonna.
Go
into
that
right
now.
A
Prometheus
is
also
written
ago,
and
it
has
an
alerting
solution,
alert
manager,
which
is
often
deployed
alongside
Prometheus.
We
absolutely
love
from
atheists
at
digitalocean.
We
currently
have
about
actually
more
than
85
instances
of
Prometheus
in
production.
We
have
a
data
center,
wide
monitoring
system
known
as
Pandora,
but
and
we
also
use
Prometheus
to
monitor
different
services
in
our
stack
and,
as
I
mentioned
before,
we're
scraping
about
1.8
million
samples
per
second.
A
So
a
lot
of
data,
the
reasons
we
love
Prometheus
are
a
it's
easy
to
deploy,
as
I
mentioned
before
it
is
written
and
go
so
you
could
deploy
it
as
a
go
binary
to
a
virtual
machine
or
if
you
containerize
it,
and
deploy
it
on
something
like
kubernetes,
it's
flexible
and
extensible.
It
has
something
called
labeling,
which
is
extremely
helpful
in
the
kubernetes
ecosystem.
When
you
have
things
that
are
deployed
in
pods,
it
also
has
a
lot
of
service
discovery
mechanisms
also
helpful,
especially
given
that
we
have
kubernetes
such
as
real
ocean.
A
A
There
are
four
core
metrics
counters,
which
is
essentially
a
monotonically
increasing
value
gauges,
a
numeric
value
that
goes
up
or
down
histograms
and
summaries
for
fairly
similar
and
refer
to
a
bucket
as
count
of
observations
using
these
four
vet
core
metrics,
as
well
as
the
prompt
QL
functions,
you
can
actually
achieve
all
of
the
four
goals
signals
as
well
as
use
metrics,
so
we
can
use
counters,
apply
the
rate
function
to
them
and
get
error
or
request
rate.
We
can
often
break
this
down
by
you
know,
type
of
request
as
well.
A
So
that
is
what
Prometheus
is,
but
now
we're
gonna
finally
get
to
the
core
of
my
talk,
which
is
our
build
instrument
and
test
and
iterate
loop
or
how
we
build
services.
So
first
thing
we
typically
do
design
a
service
submit
an
RFC.
It
gets
some
feedback
for
the
RFC.
We
then
write
our
service
and
go,
which
is
the
language
of
choice
at
digitalocean.
We
do
use
other
languages,
but
a
lot
of
our
work
is
an
NGO
and
we
have
a
go
mono
repo
as
well
I,
believe
it's
been
written
about.
A
We
have
a
lot
of
internally
shared
libraries
that
we
leverage
as
well
and
here's
kind
of
an
example
of
one
of
them,
Dr
PC,
which
is
our
shared
RPC
library.
We
love
our
shared
libraries
simply
because
we
get
a
lot
of
things
automatically
with
geo
RPC.
All
our
RPC
services
already
have
logging,
metrics
and
tracing
interceptor
incorporated,
so
we
can
actually
get
relevant,
I
guess
metrics
about
our
RPC
services,
such
as
latency
request
rate
whatnot.
A
We
also
have
traces
or
spans
that
are
sent
to
our
trace
collector
and
we
have
logs
that
automatically
get
sent
to
our
centralized
logging
system
very
helpful.
But
on
top
of
these
shared
libraries,
we
often
also
have
to
instrument
our
services,
so
we
might
have
to
explicitly
configure
our
services
such
that
logs
get
sent
to
centralized
logging
by
logging
into
syslog.
A
We
have
spans
or
our
times
trace
operations
that
are
sent
to
our
trace
collectors,
assembled
and
then
sent
to
our
tracing
back-end,
and
then
we
also
have
to
configure
our
Prometheus
metrics
there's
two
ways
we
go
about
that.
There's
of
course
the
go
client
for
Prometheus
that
we
definitely
leverage
and
then
simply
just
register
these
metrics
with
the
global
Prometheus
registry.
So
above
we
have
an
example
of
the
gauge
metric.
A
A
The
next
way
we
set
up
metrics
is
by
using
the
collector
interface.
So
the
collector
interface
is
this
structure,
but
essentially
is
you?
Can
you
can
implement
this
collector
interface
and
the
the
structure
that
you
use
to
implement?
It
typically
has
two
functions
and
the
collect
metric
is
key.
It
basically,
and
it
shows
us
that
every
time
Prometheus
scrapes
your
instance,
the
collect
function
is
called
and
metrics
are
sent
to
that
particular
channel.
A
So
using
the
collector
interface,
we
often
do
one
of
two
things.
We
often
just
build
out
a
third
party
exporter
or
we
use
a
third
party
exporter.
So
we
use
my
sequel
RabbitMQ
and
you
know
the
node
exporter
just
to
get
virtual
machine
metrics
in
a
lot
of
our
components.
So
those
are
three
exporters
we
definitely
use,
and
the
OVS
exporter
is
one
of
the
examples
of
things
that
we've
built
out
to
get
some
Sdn
metrics.
A
We
also
use
the
collector
interface
internally
within
our
code
and
here's
an
example
of
a
rate
limiter.
We
built
out
and
I'll
get
to
this
example.
A
little
bit
later.
The
key
things
to
note
about
this
is
we
actually
have
some
data
that
we're
collecting
internally
within
this
structure,
but
then
we're
exporting
and
we're
kind
of
providing
a
snapshot
of
request
rate
metrics
every
time,
Prometheus
scrapes
this
service.
A
So
once
we've
information
our
service,
we
build
a
ton
of
dashboards
seriously.
This
is
a
lot
of
dashboards,
so
the
first
dashboard
I'm
going
to
show
you
actually
links
is
our
master
dashboard
of
sorts
and
it
links
to
all
the
components
that
are
in
the
system.
This
is
for
VP
C,
so
we've
we've
got
our
state
metrics.
A
Just
you
know
what
is
up
or
down
and
in
our
system
like
the
number
of
components
that
are
up
or
down
and
we've
got
links
to
console
our
load
load,
balancer
dashboard,
our
my
sequel,
dashboard
and
one
of
our
micro
go
micro
service,
dashboards,
North
T
for
our
services
themselves.
You
can
see
that
we're
definitely
measuring
the
four
Goldie
signals
on
the
left-hand
side.
We've
got
request
rate
and
it's
split
out
by
type
of
request
rate.
A
A
We
also
have
our
use.
Metrics
we've
got
number
of
go
routines,
CPU
usage
memory,
disk
usage,
a
lot
of
these
utilization
metrics
and,
as
I
mentioned,
we
also
use
my
sequel.
So
we
also
have
use
metrics
as
well
as
request
rate
metrics
for
our
my
sequel
database.
So
we've
got
queries
per
second
and
the
number
of
my
sequel
connections
that
we
have
and
just
when
you
thought
there
couldn't
be
down
at
dashboards,
there
were
still
yet
more
dashboard.
So
we
also
have
saturation
metric
for
our
cues.
A
That
we're
using
and
saturation
I
mentioned
is
something
that's
often
a
little
trickier
for
people
to
understand.
But
hopefully
this
example
gives
you
an
idea,
so
we've
got
RabbitMQ
queue
depth
as
well
as
the
message
status.
So
what
is
kind
of
our
ratio
of
unacknowledged
messages
versus
ready
messages?
A
So
once
we
have
our
dashboard
set
up,
we're
ready
to
start
testing,
and
so
we
do
teach
three
types
of
testing
load
testing.
It's
definitely
possible
for
us
to
build
some
sort
of
load
tester,
but
we
find
that,
given
that
a
lot
of
our
services,
our
RPC
services,
we
can
simply
use,
have
a
number
of
concurrent
G,
RPC
clients
and
then
have
them,
send
requests
to
our
RPC
services
and
go
routines
to
have
a
sufficient
load
generated.
We
also
do
a
lot
of
chaos
testing,
given
that
every
component
often
is
part
of
a
larger
system.
A
So
we
like
to
see
how
failover
occurs
or
how
recovery
occurs
when
a
component
goes
down
and,
of
course
we
also
do
a
lot
of
integration
testing,
especially
on
the
networking
services
team.
We
kind
of
work
at
the
bottom
of
the
stack
are
largely
operating
on
the
software-defined
network
on
the
hypervisor,
so
it's
very
important
to
ensure
that
any
new
service
that
we
push
out
to
production
integrates
well
with
the
rest
of
the
cloud.
A
So
testing
is
very
helpful
in
that
first,
we
can
identify
key
issues.
So
here
are
some
questions
that
we
often
get
while
testing
and
you'll
actually
see
this
in
some
of
the
examples
that
I
go
through
later.
How
is
our
latency?
Maybe
if
we
have
a
latency
spike,
we
can
use
tracing
to
dig
down
further.
Is
there
a
go
routine
link?
This
is
also
something
that
has
happened
and
we
ended
up
using
a
worker
pool.
A
We
can
also
see
if
there's
a
resource
spike
with
traffic
and
use
a
different
tool,
CPU
or
memory
profiling,
to
drill
into
that.
We
can
also
tune
metrics
with
testing
so
quite
often
we'll
go
through
a
process
of
load,
testing
or
chaos,
testing
and
realize
that
a
lot
of
metrics
that
we
configured
were
actually
useless.
So
we
could
just
stop
collecting
them.
A
Conversely,
we
occasionally
realized
that
we
don't
have
enough
of
a
particular
type
of
metrics,
so
we
instrument
our
service
a
little
bit
more
and
then
something
that
I
mentioned,
that
we
loved
about
prometheus
is
the
fact
that
there
are
labels
which
allow
us
to
actually
slice
and
dice
our
data
and
different
dimensions
or
aggregate
them
across
different
dimensions.
So,
quite
often
we
also
test
and
realize
we
need
more
labels
to
get
better
introspection
into
our
services.
So
we
end
up
adding
labels
to
our
data.
A
We
also
end
up
tuning
alerts,
and
this
is
especially
useful,
I
believe
with
load
testing,
occasionally,
chaos
testing,
because
we
can
see
exactly
at
which
point
our
service
fails
and
set
up
threshold
alerts.
Based
on
that,
we
can
also
do
state-based,
alerting
just
generally
very
important
for
on-call
engineers
and
the
final
thing
no
one
really
particularly
likes
writing
documentation,
but
it's
very
important
and
so
I
think,
especially
for
the
last
two
products.
I've
worked
on
the
networking
team
at
the
tail
end
of
doing
a
lot
of
recovery,
testing
and
load
testing.
A
A
So
once
we
go
through
all
of
these
steps,
building
our
go
services,
instrumenting
them
writing
dashboards
and
testing
them.
We
like
to
iterate
and
I
think
the
best
way
to
showcase
how
we
iterate
is
just
by
going
through
some
examples,
so
that
kind
of
brings
me
for
the
last
part
of
my
talk.
So
the
first
example
I'm
going
to
talk
about
is
DHCP.
This
is
something
all
of
your
routers
do
to
assign
IP
addresses
to
network
interfaces.
A
A
So
the
main
I
know
this
can
be
a
little
complicated,
maybe
if
you're
not
familiar
with
Sdn,
but
the
main
takeaway
from
this
is
that
HP
energy
receives
customer
DHCP
traffic,
so
we
should
probably
load
test
it.
So
we
did
that
we
spent
a
ton
of
dhcp,
discover
messages
to
our
DHCP
server
and
lo
and
behold
it
fell
over.
There
was
a
spike
in
memory,
usage
and
CPU
usage,
and
then
we
managed
to
emit.
We
realized
a
couple
of
things
at
this
point.
A
Why
we
needed
more
data,
so
we
actually
had
to
drop
down
to
the
Ethernet
frame
level
for
the
DHCP
server.
So
we
could
do
a
lot
of
validation,
testing
on
some
validation
testing,
on
our
request
that
we
were
receiving,
so
we
ended
up
exposing
some
of
those
metrics,
namely
the
number
of
bytes
that
were
being
read
and
written
at
the
Ethernet
frame
level,
and
so
once
we
implemented
that
we
were
able
to
see
those
results.
When
we
load
tested
again,
we
can
kind
of
now
see,
we've
got
bytes
read
and
written.
A
Actually
none
are
being
written.
The
next
thing
we
did
is
we
realized
that
we
were
having
a
service
that
ended
up
booming
when
we
were
having
a
lot
of
load
sent
to
it
and
part
of
the
reason
for
that
is
because
we
were
spinning
off
a
new
go
routine
to
service
every
request,
and
in
that
go
routine.
We
were
allocating
memory,
so
we
ended
up
using
a
worker
pool
to
limit
this,
so
we
had
a
buffered
channel
and
we
were
basically
just
reading
off
requests
from
our
buffered
channel
to
process
requests.
A
This
ended
up
helping
a
lot
in
terms
of
resource
usage
and
our
service
falling
over.
We
still,
however,
realized
that
we
absolutely
needed
to
detect
bad
actors
and
kind
of
account
for
the
case
where
a
bad
actor
could
just
very
quickly
send
a
ton
of
requests
to
our
service
and
take
it
down,
and
that's
where
this
rape
map
collector
came
in.
A
So
we
implemented
a
rate
map
structure
it
essentially
calculated
an
X
weighted,
moving
average
of
request
rate
ought,
but
on
a
per
client
basis,
and
on
top
of
that
it
had
a
rate
map
collector
internally,
which
would
expose
a
snapshot
in
time
of
the
request
rate
distribution
across
clients.
This
is
important
to
note
become,
and
we
wanted
to
do
this
this
way,
because
there
could
potentially
be
a
cardinality
problem
if
we,
if
we
attempted
to
use
traditional
Prometheus,
histogram
metrics
for
this.
A
The
final
result
is
when
we
load
tested
again,
the
service
didn't
fall
over
and
that
client
got
rate
limited
immediately.
Okay,
so
hopefully
the
conclusion
you
can
reach
from
this
is
that
we
love
testing
our
service.
We
use
metrics
to
identify
some
resource
bikes
and
then,
as
a
result
of
using
those
metrics,
we
realized
we
needed
a
more
metrics.
We
ended
up
building
a
rate
limiter.
We
ended
up
building
using
a
worker
pool
to
limit
resource
usage
and
we
set
up
alerts
so
metrics
were
very
key
and
helping
us
identify
issues
and
ameliorate
those
issues.
A
The
next
product
I
want
to
talk
about.
Also
a
networking
product
it
was
released
earlier
this
year
is
called
VPC
or
a
virtual
private
cloud.
So
that's
a
fairly
complicated
architecture.
Maybe
the
main
takeaway
from
this
is
there's
a
lot
of
components.
There's
my
sequel.
Cluster
there's
the
rabbitmq
Claude
Hughes,
bunch
of
them
and
their
workers
reading
from
those
queues
and
also
a
bunch
of
services
sending
things
to
those
queues.
A
A
First
thing
we
noticed
is
that,
as
load
testing
started
to
increase
in
the
load
to
North
D
increased,
there
were
some
latency
issues
with
different
RPC
calls
decided
to
dive
into
that
further
using
distributed
tracing
one
of
our
favorite
tools.
We
ended
up
looking
at
the
sinc
initial
chassis
call
in
particular,
but
notice
that
we
were
actually
load
testing
so
quickly.
In
fact,
that
some
of
our
spans
are
timed,
operations
were
being
dropped,
so
we
couldn't
actually
assemble
a
full
trace,
so
we
actually
had
to
slow
the
load
tester
down
a
little
bit.
A
A
One
of
the
other
things
that
we
often
don't
think
about
is
the
fact
that
we
actually
may
not
need
a
queue.
So,
even
when
load
testing
our
queue
depth,
pretty
much
stayed
pretty
low
and
I
think
that
common
sense
usually
dictates
if
your
queue
depth
is
pretty
high
or
your
queue
tends
to
run
full,
especially
during
load
testing.
You
might
need
to
push
more
work
to
the
edge,
but
if
your
queue
consistently
remains
empty,
it's
actually
just
working
as
a
slow
load
balancer.
So
you
might
really
not
need
it
at
all.
A
There's
also,
as
I
mentioned
ton
of
components
in
BPC,
so
we
did
a
lot
of
chaos
testing,
so
our
North
T
services
were
running
in
sort
of
a
failover
mode,
so
we
induced
a
North
to
fail.
Failure
saw
what
happened.
We
dropped.
Databases
tried
to
recover,
saw
what
happened
there,
and
then
we
also
just
killed
off
a
bunch
of
South
chased
service
workers
and
tried
to
see
what
would
happen
to
our
a
queue
depth
and
also
if
other
workers
would
be
able
to
pick
up
the
slack.
A
The
result
of
this
is,
we
realized
we
needed
to
add
a
lot
alerts.
We
added
our
state-based
alerts.
You
know
just
south
to
down
north
to
down
things
like
that
or
perhaps
down
to
a
certain
number
of
instances.
We
also
add
a
threshold
alert,
so
this
was
especially
key
when
the
my
sequel
databases
were
being
dropped.
We
added
some
alerts
to
indicate
that
the
replication
was
delayed.
A
So
hopefully
you
can
see
for
VPC.
We
went
a
little
further.
We
did
both
load
and
chaos
testing
and
we
clearly
also
combined.
So
we
use
metrics
to
first
identify
that
there
was
a
problem
and
then
we
use
tracing
to
dig
down
further
and
identify
where
the
problem
was
and
fix
it,
and
then,
lastly,
we
were
able
to
configure
useful
alerts.
A
So
I
know
this
is
a
lot
of
content.
There
were
a
lot
of
graphs.
There
were
some
go
code,
maybe
that
was
unnecessary,
but
hopefully
what
you've
gained
from
this
is
that
the
things
we
care
about
and
the
things
that
we
like
to
measure
are
the
four
golden
signals
for
request,
based
systems
and
use
metrics.
We
also
like
to
instrument
our
services
as
early
as
possible.
We
could
have
I
think
we
should
do
this
earlier,
even
still
at
digitalocean.
A
Quite
often,
people
wait
a
little
too
long,
but
I
firmly
do
believe
that
instrumenting
as
early
as
possible
with
metrics
tracing
and
logging,
and
then
testing
can
help
you
identify
problems
and
especially
performance
problems.
Far
earlier
than
before.
You
release
and,
of
course,
combining
your
metrics
results
with
profiling,
logging
and
tracing
helps.
You
dig
a
lot
deeper
and
go
a
little
further
into
identifying
what
might
be
wrong.