►
Description
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
How We Gained Observability Into Our CI/CD Pipeline - Dotan Horovits, Logz.io
We all know that observability is a must-have for operating systems in production. But we often neglect our own backyard - our software release process. That was our mistake, which led us to wasting time and energy in handling failures in the CI/CD pipeline, and made our Developer-on-Duty (DoD) shifts tedious.
On this talk I’d like to share how we built effective observability into our Jenkins pipeline using intelligent data collection, dashboarding and alerting, to boost our response to failures and improve our quality of life on the way.
This talk will give practical guidance on how to improve observability into your CI/CD pipeline. Whether you use Jenkins like we do, or other CI/CD tools, you’ll learn how to augment them and reach higher productivity.
A
Glad
to
be
here
at
City,
Summit,
that's
actually
my
first,
so
I
was
one
of
the
ones
raising
hands
there
on
the
first.
One
really
excited
to
be
here
and
today,
I'd
like
to
share
with
you
how
we
gained
observability
into
our
cicd
pipeline
and,
let's
start
with
a
day
in
a
life
of
a
developer,
a
developer
on
duty,
DOD
and
the
way
it
used
to
work
for
us
every
morning
the
dod
starts
the
day
and
going
into
Jenkins
The
Graduate
project.
A
First
graduate
of
the
of
the
city
foundation
and
checking
the
pipeline
runs
from
the
recent
time.
Checking
if
anything
is
red,
especially
if
there's
a
red
master
and
sometimes
let's
admit
it,
people
forget
to
go
in
and
check
I'm
sure
it
happened
to
you
as
well,
and
if
there
is
a
failure,
then
we
need
to
go
into
the
the
runs
one
by
one
and
understand
which
failed,
where
it
failed.
How
many
failed
why
it
failed
and
when
I
say
going
to
the
different
runs,
we
have
the
back-end
pipeline.
A
We
have
the
front
end.
We
have
smoke
tests
different
different
pipelines
that
go
and
you
need
to
start
comparing
to
understand
the
patterns
across
the
runs
across
branches
and
so
on.
Also
without
any
easy
way
to
compare
with
historical
pipeline
runs
so
I
see
some
nodes
here,
so
I
guess
I'm,
not
the
only
one
who
experienced
that,
and
just
to
give
you
a
few
examples
of
questions
that
it
took
us
too
much
time
to
answer.
The
door
runs
fail
on
the
same
step.
They
don't
run
failed
for
the
same
reason
did
they.
A
The
failures
occur
on
a
specific
Branch,
maybe
you're
losing
Civic
machine.
If
something
takes
longer,
is
that
okay?
What's
the
Benchmark?
These
are
the
sorts
of
questions
that
it
took
us
too
much
time
to
answer
and
we
realized
we
need
to
improve
I'm
botan
horvitz
I'm,
the
principal
developer
Advocate
at
logs.io
at
logs
I,
always
provide
Cloud
native
observability
platform.
That's
based
on
popular
open
source
tools,
Prometheus
open
search,
Jaeger
and
so
on.
A
I
myself
I'm
an
advocate
of
Open
Source
and
communities
in
general,
and
the
cncf
in
particular,
hopefully
to
be
more
active
on
the
CDF
later
on
and
obviously
very
exciting
to
be
here
on
stage
as
part
of
kubecon.
I
am
co-organizer
of
the
local
cncf
chapter
in
Tel
Aviv,
where
I'm
based
you're
all
welcome
to
to
our
meetups
and
I,
also
have
a
podcast
open,
observability
talks,
if
you're
interested
in
the
topic
and
before
I
go
into
how
we
improved
our
CI
CD
with
the
problems
we
talked
about.
A
Let's
first
understand
what
we
need
to
improve
on
and
essentially
there
are
four
key
metrics
for
devops
performance,
they're
known
as
the
Dora
metrics,
and,
very
briefly,
it's
the
deployment
frequency
the
lead
time
for
changes
or
cycle
time
does
the
change,
failure
rate
and
the
mean
time
to
recovery,
the
mttr
they
cover
the
full
spectrum
of
pre-production
and
post-production
activities.
If
you
want
to
learn,
more
I
did
join
an
introductory
article
here
at
the
bottom.
You
have
the
QR
code.
A
A
And
as
I
mentioned,
we're
expert
at
observability,
that's
what
we
do
for
a
living.
So
for
us
it
was
very
plain
and
obvious
that
what
we're
lacking
here
in
our
case
is
observability
and
Jenkins
did
provide
some
observability.
You
could
go
into
the
pipeline
run.
As
you
all
know,
you
can
see
for
each
step
how
much
time
it
took.
So
you
could
see
that
granularity
and
we
also
wired
it
to
slack,
so
we
can
get
alerts
when
something
fails
and
so
on,
but
that
wasn't
enough
for
us.
A
We
wanted
to
find
a
way
to
Monitor,
aggregated
and
filtered
view
across
different
pipeline
runs
across
different
branches
across
machines
to
be
able
to
see
the
full
picture
on
a
scale
time
frame
with
our
own
filtering
demands.
So
we
set
out
on
this
project
in
the
engineering
with
these
four
requirements.
First,
we
wanted
a
dashboarding
and
aggregated
views
to
see
exactly
what
I
was
talking
about.
The
across
pipeline
runs
across
branches
across
machines
and
so
on.
A
So
these
are
the
four
requirements
and
now
that
you
understand
what
we
want
to
achieve,
let's
talk
about
how
to
achieve
it,
and
essentially
it
takes
four
steps:
collect
store,
visualize
and
Report
in
terms
of
the
tech
stack,
we
have
a
lot
of
expertise
with
the
elk
stack,
elasticsearch
and
Cabana,
and
now
also
with
open
search.
So
for
us
that
was
the
natural
point
to
start.
A
So
we'll
start
with
that
and
show
you
how
we
went
through
these
four
steps
and,
let's
start
with
a
collect
phase,
what
we
did
is
we
instrumented
the
pipeline
to
correct
all
of
the
relevant
information
and
put
it
into
environment
variables?
What
kinds
of
information
you
can
see
here?
The
the
branch,
the
committee,
the
machine,
IP,
the
Run
type,
whether
it's
scheduled
or
triggered
because
of
merge
to
master
or
for
push
to
Branch
or
whatever
the
build
number
anything
essentially
that
you
can
get
out
of
your
pipeline
grab.
A
It
that's
a
collect
phase
and
then
after
collect
comes
store
for
that.
We
created
a
new
summary
Step
at
the
end
of
the
pipeline,
where
we
ran
a
command
to
collect
all
of
the
pipeline
information
from
the
environment
variables
and
create
a
Json
with
all
of
this
information
and
stored
it
in
elasticsearch,
we
have
our
own,
so
it
was
a
natural
for
us
and
now
we
move
to
open
search.
A
A
Jenkins
keeps
all
the
bills
and
stores
them
on
the
Jenkins
service
on
the
Jenkins
machines,
which
could
burden
these
machines
and
the
best
practice
is
to
then
you
need
to
start
deleting.
You
need
to
limit
how
many
builds
to
keep.
How
many
days
to
keep
and
it
limits
your
your
historical
data
and
that
wasn't
enough
for
us,
we
needed
a
more
powerful
access
to
our
historical
data.
A
So
that's
on
the
stool
phase
and
after
store
it
comes
visualize,
and
now
that
all
the
pipeline
run
data
is
in
elasticsearch,
it's
very
easy
to
build,
Cabana,
dashboards
and
visualizations.
If
you
work
with
open
search,
then
with
open
search,
dashboards
and
I
want
to
recap
on
the
questions
that
I
mentioned
before
that
were
hard
difficult
to
to
answer.
Things
such
as
the
Dual
runs
fail
on
the
same
step.
The
door
runs
fail
for
the
same
reason:
how
many,
if
they
fail,
failed
on
a
specific
Branch
or
on
a
specific
machines.
A
These
questions
dictated
the
types
of
visualizations
that
we
needed
to
create.
So
I
want
to
give
a
few
examples
of
visualizations
to
answer
these
sorts
of
questions.
Just
for
you
to
tease
sort
of
similar
visualizations
that
you
may
want
to
explore
in
your
case
start
with
the
Top
Line
View
standing
the
how
stable
the
pipeline
is
for
that.
There's
the
success
and
failure
rates
overall
in
a
table
or
on
a
specific
time
range
on
a
graph
very
easy
to
just
see
in
one
bird's
eye
view.
A
A
If
you
want
to
see
the
text
problematic,
build
machines,
then
failures,
segmented
by
machine
and
by
the
way
that
was
very,
very
useful.
It
saved
us
wasting
time
looking
for
issues
in
the
released
code.
Rather
when
we
see
such
a
machine,
we
just
go.
We
kill
the
machine,
we
let
the
auto
scaling
spin
up
a
new
one,
and
then
you
start
clean
and
oftentimes
that
solves
the
problem.
A
So
environmental
issues
with
the
CI
CDs
is
a
problem
and
I'll
get
you
back
to
that
also
later,
and
the
duration
per
step,
as
I
mentioned
Jenkins,
does
allow
you
to
see
per
step
in
a
specific
run,
but
we
wanted
to
see
aggregated
view
across
pipelines
across
branches
across
machines
so
and
use
to
scale
and
to
filter.
That's
that
answer
this.
This
requirement
and
after
visualize
comes
a
report
and
alert,
as
you
recall,
before
the
dod,
the
developer
on
duty
needed
to
go
in
and
manually
check.
Jenkins
for
the
last
runs.
A
Now
you
get
beginning
of
day
start
of
day
a
report
to
your
slack,
something
like
you
can
see
here
on
the
screenshot,
with
a
link,
obviously
to
the
full
dashboard
and
with
the
snippet
of
the
dashboard
inside
the
notification
itself.
So
oftentimes,
it's
fairly
easy
to
see
already
from
the
snapshot.
If
you're
good
to
go-
and
you
can
finish
your
morning,
coffee
or
if
you
need
to
go
quickly,
link
and
click
the
link
and
start
investigating.
A
And
yeah
sorry-
and
we
can
also-
we
can
also,
of
course,
create
a
lot
on
top
of
that.
You
can
compose
any
composite
alert
based
on
the
data
that
we
collected
in
the
collect
phase.
So
you
can
ask
things
such
as
if
a
number
of
failures
sum
of
failures
goes
above
X
or
the
average
duration
goes
above
y
trigger
and
lot,
essentially
anything
that
you
can
formalize
as
a
Lucin
query,
you
can
automate
as
an
alert
and
by
the
way
this
is
done
based
on
alerting
mechanism.
A
A
So
that's
about
Jenkins
but,
as
we
know,
CI
CD
is
not
just
Jenkins
and
what
else?
If
you
recall
the
requirements,
we
also
wanted
to
analyze
flaky
tests
and
test
performance.
So,
following
the
same
process
that
I
mentioned,
we
collected
the
relevant
information
from
our
test,
run
and
store
the
data
in
elasticsearch
and
then
created
Cabana
dashboards.
A
As
you
can
see
here,
you
can
see
failure,
count,
failure
rate,
moving
averages,
failed
tests
by
Branch
over
time
and
so
on,
so
really
powerful
and
as
before.
In
addition
to
visualize,
we
can
also
create
reports
and
alerts.
So
following
the
same
flow,
one
important
point
to
mention
is
the
flexibility.
Once
you
have
the
data
in
elasticsearch
or
an
open
search,
the
team
has
the
flexibility
to
choose
which
Village
visualization
serves
the
need,
so
different
teams
create
different
visualizations
over
the
same
data
to
suit
their
needs
in
style.
This,
for
example,
is
the
Other
Extreme.
A
They
didn't
like
graphs.
They
wanted
everything
in
table
view
just
to
say
so.
That's
the
full
process,
and,
let's
summarize,
we
instrumented
Jenkins
pipeline
to
collect
all
the
Run
data
that
we
could
and
put
it
in
environment
variables.
Then
we
at
the
end
of
the
pipeline.
We
created
the
Json
with
all
that
data
and
persisted
it
to
elastic
cetera,
open
search.
Then
we
created
Cabana
visualizations.
A
On
top
of
the
data
and,
lastly,
created
reports
and
alerts
on
the
data,
collect,
store,
visualize
and
Report,
so
that's
the
beginning
of
our
journey,
but
then
what's
next,
what?
If
you
want
to
investigate
the
performance
of
a
specific
pipeline's
run?
That
was
the
following
question
and
actually
distributed
tracing
is
perfect
for
that
and
in
fact,
Jenkins
can
emit
traces
just
as
it
can
emit
logs.
So
we
decided
to
visualize
jobs
and
and
pipeline
executions
on
distributed
tracing,
how
many
no
distributed
tracing
with
a
raise
of
hands.
A
Okay,
we
have
around
half
of
the
people,
knowing
so
very
briefly
on
distributed
tracing
it's,
it
helps
pinpoint
where
failures
occur
and
what
causes
poor
performance
in
production
systems.
So
it's
not
specifically
for
CI
CD.
So
if
you
think
about
a
microservice
architecture
with
a
request
flowing
through
a
chain
of
interactive
microservices,
if
you
want
to
ask
a
question
such
as,
where
did
the
failure
occur
through
this
chain
or
where
did
the
performance
degradation
come
from
where's
the
critical
path?
A
Let's
see
how
to
add
it
to
our
CI
CD
pipeline
following
the
same
flow,
collect,
store,
visualize
and
Report,
and
let's
start
with
a
collect
phase
for
the
collect
phase.
We
decided
to
use
open,
Telemetry
collector
for
those
who
don't
know
open
Telemetry,
it's
an
open
source
that
provides
unified
collection
of
logs,
metrics
and
traces
unified
standard
way.
A
It's
a
cncf
project
actually
delivered
the
talk
about
open
Telemetry
at
cubecon,
Europe
back
in
May
and
at
the
time
open
Telemetry
was
already
GA
generally
available
for
distributed
tracing,
so
we
chose
it
and
the
way
to
do
that
is
very
simple.
You
install
the
open,
Telemetry
collector.
You
need
to
then
install
the
Jenkins
open,
Telemetry
plugin
and
then
configure
the
plugin
to
send
the
span
data.
The
trace
data
to
the
open,
Telemetry
collector
endpoint
in
otlp
over
grpc
protocol.
A
So
pretty
simple,
that's
the
collect
phase
and
after
collect
comes
store
and
for
backend
we
used
Jenkins,
sorry
Jaeger,
which
is
a
very
popular
open
source
tool
for
distributed
tracing.
It's
also
a
cncf
project
and
we
use
Jenkins
to
monitor
our
production
environment.
So
we
did
the
same
here.
We
also
have
our
own
managed
Jaeger.
So
we
use
that,
but
it
works
just
the
same.
If
you
have
your
own.
Actually,
if
you
run
it,
I
wrote
a
guide
on
how
to
install
a
jageron
kubernetes
in
production.
A
You
have
a
link
here,
if
you
find
it
useful
and
what
you
need
to
do
in
this
phase
is
to
essentially
configure
the
open,
Telemetry
collector's
exporter,
to
send
it
to
the
Jaeger
back
end,
and
that's
that's
about
it
and
once
you
have
that
after
store
comes
visualize.
The
visualized
step
here
is
very
simple,
because
Jaeger
UI
has
built-in
visualizations,
you
don't
need
to
start
composing
dashboards.
The
most
popular
view
is
the
timeline
view.
A
As
you
can
see
here,
on
the
left
hand
side,
you
see
the
indented
tree
structure
and
then,
on
the
right
hand,
side
the
gun
chart
and
it's
very
easy
to
see
the
pipeline
sequence.
You
can
see
where's
the
critical
path
you
can
see
how
much
time
each
step
took,
which
steps
ran
in
parallel
and
which
sequentially
and
if
you
only
have
a
pipeline
that
the
runtime
is
too
long
and
you
need
to
optimize
it's
very
easy
to
see
where
most
of
the
time
is
spent
and
where
to
optimize.
A
A
A
You
can
later
add
the
instrumentation
for
Maven
for
ansible
for
other
tools
to
gain
finer
granularity
into
your
traces
and
into
your
steps.
For
example.
Here
everything
that
you
can
see
in
yellow
is
Maven
so
that
what
before
was
one
big
chunk?
One
big
span
now
black
box.
Now
you
can
open
it
and
see
the
breakdown
into
the
individual,
build
steps
in
Maven
and
the
individual
details
and
lengths
of
each
one
so
very,
very
useful.
A
So
we've
seen
how
to
add
tracing
into
our
CI
CD
Pipeline
with
the
same
steps,
collect,
store,
visualize
and,
of
course
you
can
also
create
alerts
and
reports,
just
like
any
other.
It's
like
elasticsearch
like
open,
search,
very
easy,
and
what's
next
so,
if
you
recall
I
mentioned
that
many
of
the
pipeline
runs
actually
failed
because
of
problems
in
the
environment
rather
than
a
problem
with
the
deployed
code,
so
we
decided
to
add
and
monitor
metrics
from
the
Jenkins
servers
and
the
environment.
A
The
system,
the
containers,
the
jvm,
essentially
anything
that
could
fail,
irrespective
of
the
released
code
and
monitor
that
so
following
the
same
flow
starting
with
the
collect
phase
for
the
collect
we
use
Telegraph
or
production
environment.
We
also
use
that
here
it's
an
open
source,
bindflex
data
and
one
of
telegraph's
strong
suits-
is
the
rich
Suite
of
Rich
Suite
of
plugins
that
it
has.
So
you
have
a
general
link
here
for
the
plugins
directory.
I'll
mention
a
few
plugins
here,
and
what
you
need
to
do
is
two
steps.
A
First,
you
need
to
configure
Jenkins
to
expose
metrics
in
Prometheus
format.
We
use
Primitives
extensively
in
our
production
environment.
We
also
use
that
for
the
cicd
monitoring
and
that's
a
very
simple
configuration
in
the
Jenkins
web
UI,
and
then
you
need
to
install
Telegraph.
If
you
don't
already
have
one
and
configure
Telegraph
to
scrape
the
metrics
off
of
the
Jenkins
servers
with
a
Prometheus
input.
Plugin,
okay,
Prometheus
input,
plugin,
that's
what
you'd
be
looking
for.
That's
first
phase.
A
Second
phase:
as
we
said
after
collected
store
and
as
I
mentioned,
Prometheus
is
our
choice
for
production
environment.
Also,
the
choice
for
cicd
monitoring:
we
have
actually
have
our
own
Prometheus
as
a
service,
so
we
use
this
one,
but
the
same
would
work
for
any
Prometheus
or
Prometheus
compatible
back-end
store,
and
what
you
need
to
do
is
to
configure
Telegraph
to
send
the
metrics
to
Prometheus,
and
you
have
two
options
for
that:
you
can
do
it
in
pool
mode
or
in
push
mode.
A
Pull
mode
is
the
default
for
Prometheus,
in
which
case
you
configure
Telegraph
to
expose
the
slash
Matrix
endpoint
that
Prometheus
can
scrape
if
you
want
to
do
that,
you
use
the
Prometheus
client
output
plugin
and
if
you
want
to
use
the
push
mode,
use
the
HTTP
output
plugin
with
the
data
format,
Prometheus
remote
right.
So
very
simple:
that's
the
store
phase,
end
of
the
store.
Once
we
have
all
the
data
in
Prometheus,
it's
very
easy
to
create
a
grafana
visualizations
on
top
of
that
data.
A
On
top
of
the
time
series
data-
and
you
can
see
example
of
the
dashboard
here.
Obviously,
you
can
filter,
as
you
can
see
at
the
top
here,
you
can
filter
by
the
build
Tie
by
the
branch
by
the
machine
ID
by
the
build
the
build
ID
and
other
filter
so
very
easy,
and
this
is
the
example
of
the
system
system,
type
of
monitoring,
CPU
memory,
disk
usage
and
so
on.
You
also
can
monitor
the
docker
container
metrics.
A
A
If
you
can
see
here,
the
Jenkins
job
count
over
time
breakdown
by
job
status,
how
many
aborted,
how
many
failure
not
built
and
so
on
so
lots
of
information
here
and
actually
that
was
so
useful
that
our
users
asked
that
for
the
for
that
dashboard.
So
we
made
that
available
as
a
pre-built
dashboard.
They
can
install
in
one
click
because
it
was
very
useful
and
of
course,
you
can
set
alerts
on
that
time.
Series
data
on
the
on
the
metrics.
A
A
A
These
were
the
goals
that
we've
achieved
and
we
did
that
with
four
steps.
Collect
the
store,
visualize
and
put
be
collect
being,
is
instrumenting
your
pipeline
to
get
events,
the
state,
metrics,
traces
and
so
on,
store
and
visualize
according
to
the
type
of
data,
so
elasticsection
open,
search
for
the
logs
and
events.
Promises
and
graphene
for
metrics
yeah
you
get
for
traces
and
so
on,
and
then
alert
and
report
to
automate
as
much
as
possible
and
last
Point
what
we
gained
a
significant
improvement
in
our
lead
time
for
changes
dramatically
shortened.
A
A
You
have
the
link
and
the
QR
code
here
if
you
want,
and
if
you
have
any
comments
on
the
talk
or
on
the
guide,
feel
free
to
reach
out
to
me
on
Twitter
at
horvitz
or
I'll,
be
here
at
the
logs
IO
Booth.
If
you
want
to
grab
me
in
person
or
I'll,
be
on
the
virtual
Booth
later
today
on
a
q,
a
I,
don't
know
if
you
have
time
now
for
a
q,
a
so
up
to
you,
we
don't
have
so
you
can
grab
me
later,
then.