►
Description
Measuring DevOps Success with Pipeline Analytics - Chris Riley, Splunk, Inc.
Speakers: Chris Riley
Do you understand the value of “good” in DevOps? The software delivery chain is the app of apps. If your delivery chain is not up, running and performant, nothing gets deployed. But how do you measure success? Do you actually know how well your DevOps automation is contributing to your company's bottom line? Chris Riley, DevOps Advocate from will explore the practice of pipeline analytics and demonstrate how tools like Dora and Flow metrics are tools for measuring the performance of your SDLC and the impact of DevOps on your business success. Join to learn: · How pipeline analytics are a necessary practice in all high-performing DevOps environments · How to select the right metrics to measure DevOps success · And how to implement build a Pipeline Analytics strategy
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
A
Hello,
everybody
and
welcome
to
my
session
on
measuring
devops
success
with
pipeline
analytics.
Before
I
start,
I
just
want
to
say
I
know,
there's
a
lot
of
content
out
there
today
and
everybody
is,
is
probably
getting
sick
of
being
stuck
behind
their
monitor
in
their
home
offices
here.
So
you
taking
the
time
to
join
my
session.
I
really
appreciate
that
and
also
big
thank
you
to
everybody
at
cd
foundation.
Putting
these
events
together
is
a
lot
of
work.
A
A
If
you
are
at
this
conference,
then
velocity
and
automation
is
already
your
jam.
That
is
what
ci
cd
is
about,
but
let's
say
you
got
a
chance
to
drive
your
first
f1
car.
They
said
it's
yours,
you
take
it.
You
drive
it
as
fast
as
you
want
wherever
you
want,
so
you
hop
in
and
you
ask
all
right
where's,
my
helmet.
They
say
no
helmet,
no
visor,
say:
okay,
what
about
a
seat
belt,
no
seat
belt?
A
A
A
A
So
it's
not
beneficial
to
have
these
visibility
silos
where
a
handful
of
people
deeply
understand
what's
going
on,
but
nobody
else
because
anytime,
you
have
to
go
and
explain
to
another
person
in
the
broader
engineering
organization.
What
is
happening
in
your
realm?
It's
just
wasting
time.
It
doesn't
need
to
happen,
and
if
you
love
automation,
you
know
that's
why
we
automate
so
think
about
visibility
exactly
the
same
way
and
the
reason
you
should
think
about
it.
Exactly
the
same
way
is
your
software
delivery
chain
is
an
application
anymore.
A
If
you're
attending
today
and
you're
a
developer,
you
get
to
consume
this
data
in
such
a
way
that
it
reduces
the
amount
of
time
of
unplanned
work
for
you
if
you're
a
devops
engineer,
you're,
probably
the
deliverer,
the
owner
of
this
product.
Maybe
it's
a
monitoring
organization,
if
you're
the
sre
this
product
is
what
helps
give
you
more
context
for
production
incidents
and,
if
you're,
a
security
professional,
it
gives
you
more
visibility
and
confidence
in
what's
happening
along
the
software
delivery
process.
A
A
These
are
all
elements
to
ensure
that
that
velocity
that
you
have
created
is
sustainable.
You
can
maintain
it
and
you
can
tell
somebody
whether
you're
doing
a
good
job
or
a
bad
job,
which
is
the
lifeblood
of
career
growth
and
maintaining
and
justifying
why
you
do
what
you
do
I
like
to
use
this
term
called
code
to
cloud
visibility.
A
This
is
why
we
do
all
of
this,
and,
like
I
said
earlier,
we
tend
to
be
already
pretty
good
at
what
happens
in
production.
We
have
this
new
term
observability
that
represents
monitoring
distributed
in
in
modern
applications
in
production.
We're
really
good
at
that.
We're
good
at
full
stack
monitoring,
we're
good
at
incident
response.
We
can
always
be
better,
but
this
is
kind
of
a
well-defined
area
that
companies
spend
a
lot
of
energy
on
what
a
lot
of
enterprises
aren't.
A
A
A
Also,
your
software
delivery
chain
is
a
part
of
your
attack
surface
in
the
last
four
months.
I
can
count
three
dramatic
incidents
that
have
happened:
vulnerabilities
that
have
exposed
both
the
company
who
produces
the
technology
and
the
users
of
that
technology
and
all
of
their
customers.
I
know
that
sounds
very
big,
because
I
don't
want
to
throw
people
under
the
bus,
but
I
will
give
you
some
examples.
A
Another
example
the
ide
of
all
things,
the
ide,
the
first
place
where
code
is
put
together.
There
was
a
rogue
ide
released
that
was
adopted.
Unknowingly,
I
assume,
hopefully,
by
a
lot
of
mobile
application
developers.
They
built
their
applications.
Malicious
code
was
injected
at
build
time
and
they
deployed
them.
This
happened.
They
were
building
the
vulnerabilities
themselves
without
even
knowing
it
because
of
the
ide.
A
That
is
another
software
delivery
chain
attack.
Another
example
is
a
large
company
where
artifacts
were
injected
malicious
artifacts
were
injected
as
a
part
of
the
build
process.
It
went
undetected
and
those
artifacts
ended
up
in
the
resulting
application
with
customers
and
all
the
customers
now
had
a
vulnerability
that
was
exploitable.
A
So
your
software
delivery
chain
is
a
part
of
your
attack
surface.
If
you're
a
developer
and
you,
you
think
security's,
not
your
problem.
The
reason
it's
your
problem
is
it's
disruptive,
not
because
it's
your
responsibility,
but
it
will
be
disruptive.
You
want
to
build
code
if
you're
a
devops
engineer
same
story.
It's
disruptive!
A
A
So
that's
why
developers
should
care.
I
said
earlier:
we
need
to
speak
the
same
language.
It
is
not
useful
to
have
great
visibility
just
with
your
peers
that
visibility
needs
to
be
shared
and
communicated
across
the
engineering
org,
even
across
your
product
organization
and
business
value,
to
your
leaders.
Now
that
last
one
may
be
the
one
that
was
like.
You
know,
that's
hard!
A
Well,
you
know
as
a
developer,
how
many
times
you've
been
fighting
and
wanting
budget
to
do
new
things
to
get
the
new
tool
to
get
the
new
monitoring
tool
to
get
the
new
framework
whatever
it
is
testing
tool
to
help
justify
that
you
can't
just
go
to
them
and
say
that
this
is
the
best
tool
from
a
technical
perspective.
They
will
believe
you,
but
you
have
to
demonstrate
value.
A
Part
of
demonstrating
value
is
pipeline,
analytics
and
part
of
demonstrating
the
value
of
your
overall
application
is
pipeline,
analytics,
so
understanding
the
impact
of
your
features
on
your
customers
and
how
that
impacts.
The
bottom
line
of
the
business
is
also
pipeline
analytics.
So
these
are
all
tools
that
in
retrospect,
you
wish
you
were
doing
in
advance
now
the
other
thing
is
just
period.
A
How
do
you
know
you're
doing
a
good
job?
How
do
you
know
that
working
or
moving
to
refactoring
to
kubernetes
benefited
the
organization?
Do
you
really
know?
How
do
you
know
that
implementing
an
on-call
incident
response
strategy,
reduce
the
number
of
incidents
or
compress
the
time
of
recovery
from
mean
time
to
detect
to
mean
time
to
recover?
A
A
The
benefit
of
that
and
this
gets
to
the
next
point
is
the
shift
left
anything
you
can
do
earlier
in
the
software
delivery
chain
costs
the
organization
less
money,
and
so
by
addressing
technical
debt.
Earlier
on,
this
can
be
configuration
drift
operational
drift.
You
are
actually
saving
the
company
a
lot
of
money
now
again
as
a
developer.
Why
am
I
talking
about
all
this
business
stuff?
A
You
are
on
the
front
line
of
being
able
to
provide
that
visibility,
and
so
having
these
tools
in
your
best
in
back
pocket
is
not
only
good
for
the
business,
but
it's
going
to
be
good
for
your
career
and
also
reduce
frustration
for
you
in
terms
of
the
interruptions
that
you
face
while
you're
developing
and
arguably
this
whole
concept
of
shift
left
shift
left
security
shift.
Left
monitoring
shift
left
incident
response.
A
Everything
relies
on
pipeline
analytics,
so,
hopefully
I've
sold
you.
I
I
it
shouldn't,
be
a
stretch
on
to
why
you
should
do
it.
The
software
delivery
chain
is
an
application,
treat
it
like
an
application
as
easy.
As
that,
the
good
news,
it's
not
that
hard
to
implement-
and
it's
probably
something
that
you
can
start
to
approximate
or
start
with
today
with
tooling.
You
already
have-
I
I
hope
so
really.
What
is
it?
A
You're,
monitoring
your
software
delivery
chain,
you're,
creating
a
value
stream
and
kpis
for
the
entire
engineering
organization
and
you're,
also
enabling
in
automating
we
love
to
automate
audit
and
security
such
that
your
compliance
team,
your
security
team
doesn't
need
to
come
to
you.
Stop
everything
for
two
weeks
to
figure
out
what
you're
doing
you
can
instantly
point
to
what's
going
on
in
your
delivery
chain?
A
How
do
you
do
it
again,
not
very
difficult?
You
gather,
metrics
and
logs
from
the
tools
in
your
tool
chain,
so
cicd
tools,
your
release,
automation,
tools,
jenkins
and
spinnaker
are
going
to
be
one
of
the
number
one
sources
of
data,
not
the
only
source.
The
top
sources
are
your
ticketing
system,
your
ci
cd
tool
and
your
incident
response
tool
in
some
organizations
who
haven't
implemented
an
incident
response
strategy,
their
ticketing
tool
once
you've
got
the
data,
you
correlate
it
across
processes
and
teams,
and
then
you
make
it
observable,
and
that
is
a
nuance.
A
It's
not
just
a
dashboard.
You
you
send
to
somebody
you
have
to.
You
have
to
be
on
the
same
page
with
the
rest
of
the
or
engineering
organization,
and
it
almost
always
breaks
down
into
these
three
primary
use
cases
and
the?
U
cases
are
important
because
they're,
driven
by
different
people
and
the
data
is
leveraged
by
for
different
people,
so
first
is
monitoring.
A
Monitoring
is
usually
driven
by
devops
engineering
teams
leveraged
by
devops
engineering
teams.
Measure
measure
is
usually
driven
by
devops
engineering
and
development
teams
and
leveraged
by
everybody,
but
mostly
developers.
Audit
audit
is
usually
driven
by
devops
engineering
teams
and
security
teams
and
compliance
teams,
but
the
data
is
leveraged
by
sres
and
security
teams.
Most
often
those
are
the
primary
everybody
in
the
engineering
organization
benefits
from
this,
but
each
of
these
use
cases
have
different
personas
who
are
driving
it
and
leveraging
the
data.
A
So
starting
with
monitoring
monitoring
again,
your
delivery
chain
is
up.
If
it's
not
up,
then
you
can't
deploy
code.
So
this
is
for
devops
engineers
to
meet
their
service
level
objectives
for
the
delivery
chain.
We've
already
been
doing
this.
We
do
this
all
day,
long
with
applications
in
production.
These
are
your
standard
application
and
infrastructure.
A
Metrics
you're
going
to
be
looking
at
system
metrics,
like
memory,
cpu
disk,
simply
status.
Is
it
up
or
is
it
down?
You
have
red
metrics
on
the
application
layer
for
rate
error
and
duration
and
use
metrics
for
your
infrastructure,
utilization,
saturation
and
errors.
This
is
all
fairly
standard,
something
that
we
are
already
doing
today.
It
does
change
depending
on
if
you
are
using
a
sas
based,
ci
cd
tool
or
if
you
are
self-hosting
self-managed
ci
cd
tool,
like
most
people
with
jenkins
next
is
auditing.
This
is
the
one
that's
not
the
most
fun.
A
I
know
it's
not
the
most
fun.
For
me,
security
professionals
love
it.
Of
course
compliance
teams
really
love
it
developers
should
love
it.
For
that
reason,
because
this
can
be
tremendously
disruptive
when
you
are
asked
to
prove,
provide
audit
data
for
the
activity
across
your
delivery
chain.
The
metrics
I've
listed
here
are
kind
of
the
standard
activity
and
they're
all
usage
metrics.
A
Who
is
accessing
the
tools
from
where
what
are
they
doing
with
it?
This
is
especially
important
for
your
repositories
and
for
your
secrets,
management
tools,
but
also
for
your
ci
cd
processes.
You
want
to
understand
what
data,
what
metadata
is
a
part
of
your
build
processes
in
your
repos,
how
it's
being
used?
How
do
you
make
sure
that
a
secret
doesn't
matter
magically
end
up
in
a
commit
message
in
a
repo?
So
this
is
all
kind
of
standard
security
and
compliance
stuff,
and
it's
just
so.
You
have
confidence.
You
know.
A
What's
going
on,
security
professionals
will
come
knocking
on
your
door
for
this
information.
Wouldn't
it
be
nice
if
they
never
had
to
and
they
just
referenced
the
dashboard.
If
something
bad
goes
happen,
they're
still
going
to
knock
on
your
door,
but
it
would
be
better
to
have
that
information
readily
available,
so
they
didn't
have
to.
A
And
finally
this
is
the
one
that
everybody
gets
excited
about
and
I
tend
to
get
the
most
excited
about
as
well.
This
is
what
developers
really
get
a
lot
of
excitement
about,
which
is
how
do
you
know
you're
doing
well,
there's
a
lot
of
terms
in
the
industry
used
to
talk
about
kpis
for
your
delivery
chain.
We
have
two
primary
terms
that
have
kind
of
taken
over.
They
are
not
standards.
A
I
I
will
tell
you
right
now.
They
are
not
standards,
but
they
are
good
guides.
So
we
have
the
dora
metrics,
which
come
out
of
the
dora
institute,
which
at
some
point
was
acquired
by
google,
and
these
are
standard
metrics
that
came
out
of
a
lot
of
research.
They
did
prior
to
that
acquisition
and
you
have
flow
metrics,
which
also
are
driven
by
another
enterprise
software
company,
and
they
are
useful,
mostly
focused
on
velocity
of
your
delivery
chain.
All
of
these
are
good,
but
none
of
these
are
turnkey.
A
You
will
never
be
in
a
situation
where
you
can
just
take
dora
metrics
or
you
can
just
take
flow
metrics
and
implement
them.
It
is
a
bigger
process
than
that
and
that's
the
part
that
can
kind
of
get
frustrating
at
times,
because
it's
not
a
technological
problem.
It's
an
organizational
and
when
I
say
organizational
I
mean
information,
organizational
problem.
So
first
you
need
to
decide
what
you
want
to
measure
and
how
you
decide
that
is
really
important.
Developer
leader
boards,
for
example,
are
really
exciting,
but
also
very
dangerous
for
morale.
A
A
How
you
calculate
these
metrics
change
for
every
organization
and
every
tool.
There
are
slight
variations
to
all
of
these,
for
example,
starting
with
mean
time
to
recovery,
it
is
usually
calculated
by
your
incident
response
tool.
It
can
also
be
calculated
from
your
ticketing
tool
to
well
your
ticketing
tool.
So
those
are
your
two
options.
The
quality
and
the
resolution
is
going
to
depend
on
what
you
pick.
If
you
don't
have
an
incident
response
tool,
which
is
the
best
resolution
for
mean
time
to
recovery,
then
the
next
best
is
your
backlog.
A
A
Other
metrics
that
I'm
really
excited
about.
Besides
the
primary
dora,
4
metrics
are
work
in
progress,
cost
of
downtime,
and
I
especially
really
like
amount
of
unplanned
work.
The
reason
I
like
amount
of
unplanned
work
so
much
is
because
it's
a
good
indicator
of
the
amount
of
disruptions
your
development
team
is
facing.
That
is
not
allowing
them
to
build
functionality.
A
We
also
see
a
lot
of
organizations
using
this
to
audit
their
repositories
to
see
if
they
have
technical
debt
in
their
code,
repos
branches
not
being
used
or
weird
sprawl
across
their
branches.
So
this
is
a
really
exciting
area,
but
it
needs
guidance
and
caution
when
you
start
to
decide
what
you
should
measure.
A
So,
let's
get
in
in
the
next
five
minutes
on
how
to
do
this
with
jenkins
in
spinnaker.
Both
are
very,
very
similar
with
different
twists.
Both
have
pros
and
cons,
for
example,
jenkins,
tends
to
be
the
most
verbose
when
it
comes
to
logs
there's
a
lot
of
data,
but
that
can
also
be
a
problem
when
it
comes
to
storage
and
retention
with
your
logging
tool,
depending
on
what
tool
you
are
using.
A
So
the
first
step
is
to
get
the
data
there's
a
lot.
There's
there's
three
primary
ways
to
do
this.
A
lot
of
it
is
going
to
hap
depend
on
if
your
logging
tool,
if
your
visualization
tool
has
a
native
integration
that
it
that
is,
of
course,
the
first
most
beneficial
place
to
go.
If
you
have
a
native
integration
for
jenkins
and
spinnaker
great
and
they
exist,
if
you
do
not,
you
need
to
get
the
data
one
place
to
start
is
logs.
Benefit
of
logs
is
great
for
bossity
only
with
logs.
A
Sonar
cube.
The
data
that
comes
into
jenkins
logs
with
sonar
cube
test
is
very
good,
so
to
best
visualize
sonar,
cube
activity
and
test
results.
You
want
to
get
it
through
jenkins,
logs
you're,
not
going
to
get
it
through
the
rest.
Api.
The
rest
api
exists,
but
you're
not
going
to
get
it
via
that
path,
so
logs
our
greatest
verbosity,
but
also
increase
the
complexity
of
building
your
dashboards
and
increase
the
volume
of
data
rest
apis
are
really
good,
typically
only
spinnaker
and
is
going
to
be
used
for
rest,
apis
and
web
hooks.
A
A
The
bad
news
is,
you
have
to
pick
one.
You
can
reconcile
this
data
with
various
types,
various
tools,
but
you
really
need
to
pick
one
and
stick
with
it.
Most
organizations
go
with
logs
next
is
deciding
what
to
measure,
and
that
is
what
I
already
was
indicating.
What
you
pick
to
measure
and
what
it
means
to
you
is
very,
very
critical,
first
step
in
understanding
how
you're
going
to
do
it
and
it
isn't
trivial.
A
Like
I
said,
mttr
is
calculated
in
different
ways
in
different
tools,
and
you
need
to
understand
that
and
it
can
be
interpreted
in
different
ways
as
well.
Then
you
build
the
dashboards,
that's
the
fun
part.
So
once
you
have
the
data,
you
know
how
you're
going
to
calculate
the
metrics
that
you're
interested
in
you
build
the
dashboards,
not
terribly
complex.
A
Typical
dashboards
are
a
time
series
graph
historically
grouped
by
the
entire
engineering
org.
Then
projects
that's
going
to
be
different.
Based
on
your
organization,
then
there's
usually
almost
always
a
trending
up
and
down
over
a
time
period.
So
it's
day
month
week
year,
are
you
trending
up
in
in
mean
time
to
recover?
Are
you
trending
down
in
mean
time
to
recover?
Are
you
trending
up
in
velocity
or
you're
trending
down
in
velocity
over
that
time?
Almost
always
that
is
the
configuration
across,
especially
the
kpis
when
it
comes
to
audit
and
compliance.
A
This
one
is
jenkins,
focused
on
build
trends
over
time
and
build
times
which
is
really
interesting,
because
I've
seen
many
organizations
use
this
to
optimize
build
times.
The
one
above
is
auditing
compliance
being
able
to
see
who's
using
the
tool
and
how
they're,
using
it
not
terribly
complex
once
you
get
there
start
small
start
very
small
refine
and
expand.
When
I
say
start
small,
I
mean
one
metric
pick
one
metric
from
dora
or
flowmetrics,
and
do
that
and
do
that
well
and
socialize.
It
then
expand
from
there.
A
There
are
a
lot
of
considerations
you
have
to
make
that
are
not
technology,
and
I
know
this
is
the
point
where
you're
like.
Why
aren't
you
showing
me
implementation
in
code
chris,
well,
one
because
it's
tool
specific?
So
if
you
want
to
see
that,
I
need
to
know
what
tools
you're
using
but
two
there's
a
lot
of
technology
considerations
that
are
not
the
technology
itself.
A
First
is
data
resolution.
Your
resolution
increases
as
you
correlate
two
tools
together,
repo
to
ci
cd
jira
or
your
backlog
to
your
ci
cd
tool
sprawl.
If
tool
sprawl
is
a
problem
in
your
organization,
you
need
to
focus
on
specific
tools.
You
can
actually
use
this
as
a
lever
to
get
people
to
adopt
tools
correctly.
A
It's
a
crazy
thing,
so
simple,
but
if
you
don't
have
good
naming
conventions
for
your
teams
and
your
projects
becomes
very
complicated.
When
you
look
at
your
your
builds
and
your
pipelines
to
understand
what
applications
are
being
deployed,
what
teams
are
doing
it
and
correlate
that
data
in
a
meaningful
way?
A
Thank
you
so
much
30
minutes.
I
have
a
lot
more
to
talk
about
this
topic.
So
if
you
want
to
talk
more
about
pipeline
analytics,
please
reach
out.
Let
me
know
let's
discuss.
This
is
a
topic.
That's
really
important
to
me
and
I
guarantee
you
in
the
next
year
is
going
to
become
more
and
more
common,
enjoy
the
rest
of
the
event
and
thanks
for
attending
my
session.