►
Description
Delivering new software features faster, while making sure that services are solid and secure is a key differentiator for every business today. Continuous Delivery (CD) is critical to business agility and how teams deliver value. Nevertheless implementing continuous delivery practices remains challenging.
The Continuous Delivery Foundation seeks to improve the world's capacity to deliver software with security and speed. The end user council provides the opportunity for any organization to accelerate their CD adoption efforts and bring about organizational change. The first challenge the End User Leadership Forum will discuss is Measuring DevOps. Join the forum to learn from End User industry leaders how their organizations are measuring and evaluating software delivery performance.
A
You
find
things
that
you
want
to
measure.
Suddenly
you
want
to
start
measuring
everything
and
metrics
can
end
up
becoming
the
tail
that
wags
the
dog.
So
one
of
the
things
I
really
want
to
focus
on
this
conversation
is:
how
do
you
keep
in
mind
the
in-state
or
what
are
you
driving
towards?
Why
are
you
measuring?
Why
is
it
important
to
your
organization?
A
What
are
you
actually
hoping
to
accomplish
because
you
can
measure
any
number
of
things
and
the
the
time
it
will
take
to
get
good
measurements
for
all
those
different
dynamics
of
measurement
will
take
a
long
time,
so
it's
important
that
we
be
able
to
focus
on
goals
desired
in
states
and
how
metrics
can
inform
the
conversation
or
how
metrics
can
force
you
to
start
asking
questions
about
how
things
are
set
up?
How
is
your
organization
set
up
for
success
or
not?
A
What
kind
of
technology
platforms
are
you
using
and
can
metrics
be
a
useful
tool
to
help
you
make
decisions
about
your
technology
choices?
Can
metrics
help
you
figure
out
how
you're
allocating
people
and
and
resources,
and
does
it
give
you
insight
into
your
existing
processes?
Do
those
need
to
be
changed
and
some?
So
these
are
all
the
sorts
of
questions
that
I
hope
to
get
into,
because
whenever
you
start
talking
about
devsecops
and
continuous
delivery,
yes,
technology
and
tools
are
very
important.
A
A
How
do
you
make
sure
they
work
to
the
best
of
their
capability
and
that
you're,
frankly
getting
out
of
your
own
way?
So
that's
that's
kind
of
the
overview
I
want
to
establish
here.
So
we
can
figure
out.
You
know
there
are
any
number
of
resources
online
about
how
to
measure
or
what
to
measure.
I
really
want
to
push
into
the
conversation
of
you
know.
A
What
does
dora
plus
plus
look
like
what
is
the
importance
and
relevance
of
projects
like
the
four
keys
project,
for
example,
and
we
can
get
into
that
discussion
as
well,
but
anyway,
you
don't
want
to
hear
from
me.
You
don't
hear
from
the
expert,
so
I
wanted
to
use
this
as
an
opportunity
to
bridge
to
our
esteemed
guests
and
why
don't
we
start
with
dina
portman
dina?
How
are
you
today.
B
B
Okay,
I
am
dina-
and
I'm
here
today
to
talk
to
you
about
the
four
keys
project
and
measuring
devops
success
so
to
measure
devops
success.
First,
we
have
to
agree
on
what
devops
is
and
there
are
a
lot
of
different
definitions,
and
this
is
the
one
that
we
have
up
on
our
cloud:
the
google.com
site
and
it's
an
organizational
cultural
movement
that
aims
to
increase
software
delivery.
Velocity,
improves
service
reliability
and
build
shared
ownership
among
software
stakeholders.
B
B
If
you
had
a
really
big
outage
last
week-
and
it
was
all
over
the
news
and
your
vp
is
really
angry-
then
you're
going
to
have
this
like
really
outsized
understanding
of
what
your
change
fail
rate
is,
that
might
not
actually
be
true.
Also,
it's
not
automatic.
So
we
see
survey
fatigue
and,
if
you're,
trying
to
like
measure
this
over
time,
then
people
are
probably
going
to
drop
off
pretty
quickly
and
you're
going
to
have
diminishing
returns
from
your
surveys.
B
So
that
is
why
we
started
thinking
about
using
systems
data.
So
it
helps
alleviate
the
bias.
It
makes
it
a
little
bit
easier
to
measure
over
time
and
it
exposes
bottlenecks,
because
when
we
really
drill
down
into
the
systems
data,
we
can
start
to
see
things
not
just
that.
You
know
early
time
for
change
is
really
long,
but
we
can
see
that
we
had
like
a
day
before
we,
you
know
there's
a
good
day
from
our
first
commit
to
merging
it
into
our
main
branch.
B
B
So
if
you're
using
a
new
like
change
approval
process-
and
you
want
to
know
whether
or
not
this
is
you
know
pushing
the
organization
forward
in
the
ways
that
you
care
about,
then
you
can
use
the
metrics
to
you
know
to
benchmark
your
your
performance
now
and
then
you
know
after
the
experiment's
been
going
on
for
some
time
and
you
can
say
like.
Oh,
this
has
actually,
you
know,
moved
the
dial
this
much
or
this
much.
B
The
cons
is
that
it's
it's
more
difficult
to
set
up
than
a
survey
and
there
are
storage
and
engineering
costs.
So
that
is
why
we
started
the
four
keys
project.
This
is
an
open
source
project
you
can
get
to
it
at
bitly,
slash
dora
four
keys
is
on
github
and
we
wanted
to
automate
some
of
this
data
collection
and
parsing,
because
the
great
thing
is
that
you
have
all
of
the
data
that
you
need
to
calculate
these
four
key
metrics.
B
So
from
there
we
have
our
derived
table,
so
that
first
table
is
just
is
just
everything
and
for
there
we
split
out
our
tables
using
scheduled
queries
and
then
that
feeds
into
our
dashboard
so
digging
into
the
tables.
Our
raw
events
table
should
have
every
single
devops
event.
That
happens
because
we
don't
necessarily
know
yet
at
this
point
like
what
we're
going
to
need
and
how
all
of
these
things
fit
together
and
influence
each
other.
B
So
everything
goes
into
the
raw
events
table
and
then
we
have
our
scheduled
queries
to
extract
out
and
kind
of
like
abstract,
out
changes,
deployments
and
incidents.
So
you
know
your
your.
The
events
might
show
like
your
deployment
started.
Your
deployment
is
in
progress,
your
deployment
failed
or
your
deployment
was
successful.
So
from
there
we
we
pull
it
out,
and
then
we
have
like
just
one
row.
One
deployment
row
for
each
of
like
those
multiple
events
and
getting
this
right
is
actually
the
challenging
part.
B
This
is
like
where
your
own
business
operations
and
logics
has
to
come
in,
like
you
have
to
understand
what
you
know,
what
does
a
successful
deployment
mean
for
our
organization?
You
know
which
environment
are
we
pushing
it
to
like
how
much
traffic
are
we
doing
canary
rollouts?
And
so
once
you
really
like
nail
down
what
your
changes
are,
what
your
deployments
are
and
what
your
incidents
are,
the
metrics
just
follow.
B
B
But
when
you
start
to
actually
like
look
at
the
data,
then
you'll
see
that
actually
those
30
deployments
happen
on
two
different
days.
You're
only
deploying
bi-weekly
not
daily,
and
this
is
why
you
have
to
you-
have
to
count
like
the
days
that
have
deployments
rather
than
the
number
of
deployments.
The
number
of
deployments
is
not
as
useful.
We
want
to
know
frequency
okay,
so
next
we
have
lead
time
to
change,
which
is
the
amount
of
time
it
takes
a
commit
to
get
into
production.
B
And
for
that
we
need
our
deployments
and
we
need
the
list
of
all
the
changes
in
the
deployments
so
that
we
can
compare.
Those
timestamps
important
note
is
that
for
all
of
these
metrics
we're
using
median
because
the
they're,
not
normal
distributions
and
averages
tend
to
you,
know
you
when
you
have
outliers,
your
averages
can
be
like
wildly
skewed
and
we're
trying
to
get
at
like
what
the
typical
experience
is,
what
like
the
general
activity
is
rather
than
the
average,
which
you
know
when
you
take
an
average.
B
You
end
up
with
a
number
that
maybe
maybe
never
happened,
so
we
want
the
median
and
in
the
dashboard
in
the
for
keys
project
dashboard.
We
we
have
these
little
badges
that
we
color
coded
according
to
the
door
research
and
in
this
case
a
median
lead
time
to
change
of
a
week,
is
medium
performance
and
so
yellow
badge.
B
Okay,
next
is
change
fail
rate,
which
is
the
percentage
of
deployments,
causing
a
failure
in
production.
So
we're
talking
about
rates,
we
need
a
denominator.
We
need
a
numerator.
Our
denominator
is
the
total
number
of
deployments,
and
our
numerator
is
the
total
number
of
incidents.
So,
ideally,
you
would
know
which
deployments
caused,
which
incidents
this
data
can
be
sometimes
hard
to
get
depending
on
the
systems
that
you're
using.
Sometimes
it
can
be.
B
B
Sometimes
that
data
is
not
available,
in
which
case
we
would
default
back
to
the
time
that
the
incident
was
detected.
The
reason
we
care
about
when
the
incident
was
caused
versus
detected
is
because
all
of
these
metrics,
like
we're,
considering
how
the
the
user,
the
end
user,
the
customer
is
experiencing
this.
B
Okay,
this
is
our
dashboard
four
keys
dashboard
around
the
corner.
We
have
these
like
running
daily
logs,
that's
not
in
the
research.
The
reason
why
we
have
that
there
is
this
hypothesis
that
it
would
be
useful
to
have
like
an
early
check
on
the
way
your
performance
is
trending
as
you're
doing
these
different
experiments
to
try
to
improve
your
4k
metrics,
which
we
have
right
down
the
center
there.
You
know
categorized
via
the
the
door,
research
and
anti-patterns
okay.
B
It's
important
to
use
this
to
to
improve
your
own
performance.
When
we
start,
you
know,
comparing
teams,
it's
not
always
super
useful
because
they
have
different
constraints,
different
systems,
different
tech,
debt
and,
and
also
like
you
know,
one
of
the
really
important
things
that
we
see
in
the
research
is
like
how
important
psychological
safety
is
so
like
when
you
start
like
sticking
people
on
each
other,
it
doesn't
turn
out
very
well.
I
like
to
think
of
it.
B
So
there's
I
think
it's
it's
goodheart's
law.
When
a
measure
becomes
a
target,
it
ceases
to
be
a
good
measure,
so
the
metrics
are
not
the
goal.
The
goal
is
to
improve
your
performance,
not
to
improve
your
metrics.
The
metrics
should
reflect
your
performance,
but
when
we
get
so
obsessed
with
the
number
we
we
can
start
to
do
things
that
are
not
desirable.
B
That
are,
you
know,
like
kind
of
skirting
the
issue
that
are
frankly
cheating
and
then
we
we
miss
the
opportunity
to
actually
improve
our
performance
and
that's
a
shame,
so
you
know
like
a
good
way
to
do.
This
is
to
have
very
like
process
oriented
goals
like
we're
gonna
like
try,
this
new
system
and
we're
gonna
do
xyz
and
then
you're.
You
are
like,
through
that
time,
measuring
your
your
success
on
on
this
experiment
of
like
saying,
did
we
do
these
things?
B
Are
we
doing
it,
and
then
you
use
the
four
key
metrics
to
determine
whether
or
not
that
was
a
good
strategy,
and
so
the
accomplishment
is
the
testing
of
the
hypothesis.
In
that
case,
it's
actually
really
useful.
If
you
find
that
it
was
a
bad
strategy,
because
then
you
can
share
that
information
with
other
people
in
your
company
in
your
organization
and
maybe
they
can
avoid.
You
know
going
down
the
same
path.
We
save
everybody
time
we
save
everybody
money
and-
and
that
is
yeah.
That
is
how
we
like
to
approach
using
the
metrics.
A
Hey
that
that
was
great,
and
I
really
appreciate
the
listening-
the
annie
patterns
there,
because
I
have
pretty
projects
where
once
metrics
became
the
goal,
it
led
to
a
whole
series
of
just
bad
decisions
around
how
to
game
the
metrics
to
our
advantage
and
your
points
about
psychological
safety
are
spot-on
because,
as
soon
as
this
becomes
some
sort
of
performance
management
issue
it,
it
ceases
to
to
result
in
the
kinds
of
discussions
that
you
want
to
have.
So
thank
you
so
much
for
for
pointing
out
both
of
those
anti-patterns.
A
That's
those
are
really
key
things
to
to
remember,
and
thank
you
so
much
for
the
for
the
presentation.
Nick
pinston.
Thank
you
for
being
here.
How
are
you
today
not
too
bad?
Thank
you
very
much
excellent.
How
would
you
like
to
introduce
yourself,
maybe
tell
us
a
bit
about
your
background
and
and
how
you
think
about
metrics
of
devops
in
general
and
and
how
it
ties
into
overall
strategy.
C
Yeah
yeah,
absolutely
so
hello,
everyone,
my
name's,
nick
benson.
I
lead
cloud
engineering
and
I
also
head
up
to
cloud
coe
in
our
enterprise
cloud
computing
here
in
fidelity
and
again
when
we
think
of
transformations
and
and
digital
transformations,
as
they
alluded,
you
know
it's
a
core
part
of
really
driving
that
transformation.
C
You
know
really
mature
in
relation
to
agile
ways
of
working,
and
then
you
have
all
these
teams
in
the
middle,
which
you
know
are
on
that
journey.
So
one
of
the
challenges
we
you
often
see,
then,
is
how
do
you
give
that
guiding
light
to
those
teams
that
is
uniform
in
relation
to
driving?
That
improvement?
C
You
know
and
dina
talk
some
of
those
aspects
and
for
for
what
we
see
is,
and
particularly
what
I've
seen
in
my
own
career
is
that
you
know
the
door
of
metrics,
and
you
know
if
we
put
in
availability
on
on
top
of
those,
you
know
are
really
great
enablers
for
driving
that
culture,
and
particularly
you
know
in
the
work
that
I've
been
doing.
You
know
there's
two
kind
of
key
areas.
C
You
know
I
see
really
impactful
one
is
driving
metrics
from
output,
driven
metrics
to
outcome,
driven
metrics,
and
what
I
mean
by
that
is
that
these
metrics
are
even
under
I
t
related
metrics
they're,
also
business
impacting
metrics,
which
is
often
a
big
shift
right,
because
you
see
a
lot
of
it.
Metrics
is
for
I.t
in
the
sit
rit
where
the
israeli
transverse,
the
organization,
so
when
we
think
about
business
agility,
these
really
have
a
direct
impact.
So
when
we
think
about
you
know,
you
know
we
wanted
to
get
fast
feedback.
C
We
want
to
be
able
to
pivot.
On
that
feedback.
We
want
to
experiment
more.
You
know
you're
going
to
want
to
invest
in
the
capabilities
that
really
enhance
your
deployment,
frequency
and
say
your
lead
time,
for
example,
and
that's
really
powerful
to
have
that
from
an
organizational
perspective.
Another
lens
is
around
the
simplicity.
I
think
john.
You
mentioned
it
on
your
intro
there
there,
because
there
are
a
set
of
four
metrics
or
five
if
you
attack
on
availability.
C
So
you
know
it
really
helps
to
you
know,
articulate
it
easily
across
an
organization.
You
don't
have
this
bag
of
metrics.
That
you
know
are
some
big
pyramid
and
metrics
that
you
have
to
transverse
to
get
to
the
outcome.
This
simplifies.
It
means
no
matter
what
your
domain
is
in
a
very
large
organization.
C
C
Yeah
and
I
think
they
provide
that
guiding
like
because
it
is
difficult
right,
there's
and
again
when
you
think
of
that
spread
and
spectrum
of
teams
and
where
they're,
on
a
journey.
How
do
I
know
I'm
moving
in
the
right
direction
and
having
that
holistic
approach
from
an
organization
really
uniforms
it
right
and,
and
you
know,
dean
you
were
mentioning
about,
you
know
being
able
to
tell
your
teams
and
be
able
to
share
those
experiences
it
you
create
that
kind
of
common
language,
then
based
off
that.
A
You
know
one
thing
that
comes
up
immediately
as
I
look
at
the
video.
I
noticed
that
both
of
you
have
much
better
lighting
than
I
do
so.
I
need
to
like
take
a
lighting
tutorial
from
you,
so
I
can
not
look
like
a
zombie
in
these
things
anyway.
Be
that
as
it
may,
getting
back
to
the
topic
at
hand.
Thank
you
for
for
for
for
chiming
in
and
giving
us
a
bit
of
a
background
on.
A
You
know
where
you're
coming
from
and
sort
of
the
goals
that
you're
striving
for,
and
thank
you
for,
I
guess
for
the
dora
plus
one
when
you
think
of
the
the
five
metrics
when
you
think
of
availability
yeah.
So
there's
a
there's.
There
are
a
few
questions
that
have
been
popping
up
into
the
chat,
but
I'll
go
through
those
in
a
second.
But
I
guess
what
I
wanted
to
start
off
with
you
know
after
we've
had
the
introductory
introductory
period
period
is
why
don't
we
start
with
something
fun?
A
Why
don't
we
talk
about
a
time
where
a
metrics
project
didn't
go
so
well?
You
know
not
naming
names,
of
course,
because
this
is
a
blameless
post-mortem,
but
what
what
are
some
things
that
you've
seen
that,
where
you
know
the
the
best
practices
you
talked
about
earlier
weren't
followed
and
what
were
the
outcomes
as
a
result,
and
I'm
assuming
that
for
everyone?
Listening,
I'm
assuming
that
everyone
here
has
read
measure
what
matters,
or
at
least,
are
aware
of
this.
A
The
subject
matter
of
measure,
what
matters
measuring
outcomes
versus
outputs
and
thank
you
for
delaying
that
nick,
especially
as
it
ties
into
things
like
key
metrics
and
and
how
you're
trying
to
drive
towards
that.
But
what?
What's
an
instance
from
that
you've?
Seen
from
your
background,
where
you
know
things
didn't
align
the
way
they
should
have
and
and
how
did
that
happen,
and
that
either
of
you
can
take
that
if
you
want.
C
C
C
You
know
if
you
really
want
to
get
that
full
adoption.
You
know
comparison.
Is
you
know
it's
it's
gonna,
you
know,
gonna
really
hinder
you
in
relation
to
that
movement
right.
You
don't
want
teams
that
are
beginning
the
journey
being
compared
to
some
team
that
have
been
out
in
the
cloud
for
the
last
couple
of
years.
So
that's
definitely
you
know
when
we
think
about
that
adoption.
A
D
B
So
we
just
see
people
like
enforcing
the
same
wrong
decision
over
and
over
again,
because
because
you
know
the
they'll
get,
they
feel
like
they'll
get
in
trouble
or
like
people
will
get
angry
at
them
like
these.
Are
your
co-workers,
like
you,
you
you
know
made
it
so
that
I
had
three
bad
qualities
this
week
and
now
I'm
have
to
do
this
like
other
training
and
I'm
not
going
to
get
my
bonus
yeah.
It's.
A
Yeah,
it's
very
difficult.
Actually
that
leads
to
the
question
of
like,
and
I
would
put
that
in
the
category
of
humans
going
to
human,
because
the
question
is
like:
is
it
possible
to
create
a
system
of
metrics
without
introducing
the
system
of
gamification,
or
I
guess
a
better
way
to
phrase
that
would
be
you
know?
Gamification
is
something
that
is
used
on
a
regular
basis.
B
So
you
it's
you
go
out
and
you
make
more
steps
like
you.
Don't
it
like?
What
do
you
abuse
it
like?
Probably
not
like
you're,
just
going
to
take
more
steps,
it's
great
and
then
what
we?
What
we
saw
like
we're
talking
to
a
company
recently
that
did
like
devops
gamification
and
one
of
the
things
that
I
liked
about
the
way
that
they
approached
it
was
they
they
gamified
these.
These
lower
level.
B
Events
like
like
build
time
and
all
these
things
that
like
are
not
the
dora
metrics,
but
are
the
things
that
like
if
you
improve
the
door
and
metrics,
will
improve.
So
it's
it's.
Basically,
having
your
the
thing
that
you're
you're
measuring
your
success
be
be
different
related
to,
but
different
from
the
thing
that
you
are
directly
gamifying.
C
I
think
when
you
think
about
the
gamification,
even
at
a
micro
level,
again
there's
the
risk
of
comparison
and
not
having
the
context
right.
So
if
every
team
is
the
same
and
doing
the
same
thing,
fine,
but
you
know
when
you
think
of
organizations,
that's
not
the
case.
So
even
you
know
when
you're
driving
those
practices
and
the
context
can
be
slightly
different
right.
I
can
be
a
platform
team
or
an
api
team
or
hey.
C
I
write
cli's,
they're,
very,
very
different
and
therefore
you
know,
even
at
the
micro
level
it
you
know
so
you're
driving
a
risk,
then
the
behavior
you're
going
to
instill
can
be
a
negative
behavior
and
ultimately
not
drive
that
cultural
change.
And
that's
when
you
know
when,
when
I
why
I
mentioned
the
cultural
change,
you
know.
Fundamentally,
you
want
people
to
have
transparency
and
openness
in
an
organization
and
to
be
okay
with
showing
red,
as
well
as
green
and
being
okay,
to
see
the
shift
and
it's
the
trend,
and
you
know
to
be
you
know.
C
Sometimes
you
go
from
red
to
green
and
you
get
the
green
and
then
the
metric
changes.
You
know
I'm
a
huge
fan
of
one
metric
that
matters,
the
omtm
concept,
where
a
team
like
as
part
of
a
transformation
or
anything
focuses
on
a
single
thing.
It
goes
green,
but
then
it
goes
amber
again
because
the
you
know
the
line
has
shifted
again
for
that
team.
A
Yeah,
thank
you
for
mentioning
that
you
know,
and
you
brought
another
interesting
point
when
you
talk
about
you
know
when
you
mentioned
things
that
we
should
be
measuring
and
I
started
thinking
around,
how
do
you
ultimately
form
a
consensus
around
the
the
things
that
you're
going
to
track?
So
you
know
when
you,
when
you
think
about
all
the
stakeholders
in
sort
of
a
modern
software
context
or
modern
delivery
team
context,
you've
got,
I
mean
you've
got
the
developers
themselves.
You've
got.
You
know
various
executives
with
visibility.
You've
got
your.
A
I
don't
know
central
teams.
You've
got
the
continuous
your
continuous
integration
teams
themselves.
Then
you
have
like
delivery
teams
out
on
the
periphery
of
that,
then
you
have
like
others
like
legal.
You
know
risk
management,
a
bunch
of
other
people
that
have
a
stake
in
making
sure
that
things
are
working.
You
know,
as
expected,
how
do
you
bring
together
such
a
diverse
set
of
stakeholders
and
decide
on?
A
These
are
the
things
that
that
matter
to
us-
and
these
are
the
things
that
we're
going
to
use
to
measure
success
or
progress.
I
guess
measuring
progress
is
better
than
measuring
success,
because
success
automatically
tilts
us
towards
the
the
realm
of
you
know:
metrics
good,
these
metrics
good,
those
metrics
back,
but
you
know
how
do
you
bring
a
cross
or
how
do
you
bring
together
such
a
diverse
group
of
people
to
decide
on
like
the
right
strategy
to
to
go
forward
with.
C
B
What
we
see
like
has
you
know
like
worked
well
in
in
organizations,
is
communities
of
practice,
so
that
is
definitely
a
good
approach.
Then
there's
like
you
know,
sometimes
people
do
like
proof
of
conceptual.
You
have
like
one
person
in
each
of
these
roles
that
like
come
together,
join
a
team
and
then
they
they
break
off
and
and
then
like
try
to
like
spread
the
word
into
other
teams.
That
has
some
success,
not
generally
as
consistent
as
communities
of
practice.
B
B
If
somebody
has
a
preconceived
notion
that
they're
the
best
in
the
world
they
they
might
not
take
that
very
well
and
it
can
be,
it
can
be
good
to
have
like
a
neutral
party
like
facilitating
this
meeting,
because
then
you
can
start
to
like
have
like
a
blameless
culture.
You
know
everyone's
on
their
best
behavior,
because
there's
a
facilitator
who's
joining
the
people
together
and
like
trying
to
show
them
like
we
all.
E
C
No,
you
know
to
add
john
there.
I
think
you
know
why
are
we
doing
it
in
the
first
place?
Right
start
with
the?
Why
yeah
and
you
know
how
into
what
follows
later,
I
think
it's
important
that
everyone
is
on
the
same
page
as
the.
Why
we're
focused
on
this
and
once
you
have
that
baseline,
the
conversation
is
a
little
bit
easier.
C
You
know,
there's,
obviously
lots
of
different
dynamics
in
relation
to
rolling
it
out
and
driving
that
adoption,
and
you
know
particularly
elements
of
culture
and
and
the
whole
change,
but
starting
with
the
y
is,
I
think,
a
critical
bedrock
to
get
everyone
aligned
to
what
we
want
to
achieve,
and
I
think
you
know
too
is
when
you
think
about
those
outcomes
demonstrating
the
outcome.
You
know
again,
if
we
just
want
to,
we
don't
want
to
just
throw
it
out
to
the
whole
organization.
C
How
is
this
going
to
work
and
demonstrate
it
working,
there's
nothing
more
powerful,
I
think
than
seeing
something
adding
value
and
it
works
end
to
end
it's
not
just
some
powerpoint
or
some
you
know
slide
somewhere.
That's
saying
this
is
a
great
idea
and
you
know
everyone
else
is
using
it,
so
we
should.
A
C
Right,
you
know,
and
again
the
data
helps
you.
You
know
going
back
to
the
outcome.
What
are
we
trying
to
achieve,
and
you
know
dispel
of
the
opinions-
and
you
know
I
just
saw
one
of
the
questions
there
on,
I
think
talking
about
the
practices.
It
then
allows
you
to
focus
on
what
practices
can
enable
that
change.
You
know
without
again
being
opinionated.
Oh
tdd
is
an
awesome
idea,
troop
development.
B
One
of
the
things
that
I
really
love
about
the
dora
research
is
because
there's,
like
you,
know
thousands
of
metrics
that
you
could
be
looking
at
and
right.
You
know.
Often
you
don't
know
like.
We
use
a
lot
of
things
that
are
just
like
proxies
for
other
things,
and
we
don't
know
if
they
actually
correlate
or
if
they're
actually
predicted.
We
just
like
think
maybe
they
are
and
having
something
to
look
at
and
say
hey.
This
has
been
validated
via
research.
B
A
B
B
C
Yeah
and
I
think
the
adword
john
we've
seen
a
lot
of
success
with
community
practices,
the
tacit
knowledge
spread.
You
know
things
that
are
often
buried
in
a
team
or
buried
in
someone's
mind.
You
know
to
be
able
to
share
and
generate
that
sense
of
community,
which
is
you
know
again
fundamental
when
you
think
of
this
chain.
These
type
of
changes
that
we
want
to
initiate.
C
It
really
helps
spread
that
knowledge
and
again,
when
you
think
of
best
practice-
and
you
know
it
kind
of
then,
a
lot
of
the
best
practices
start
filtering
up
through
teams.
Without
you
know,
being
hardcore
standards
being
pushed
straight
down,
you
often
see
the
best
things
bubbling
up
and
creating
those
connections.
You
know
you
think
about
the
everyone's
on
video
today,
you
know
with
the
pandemic.
C
You
know
how
do
you
you
know
creating
those
connections
are
even
more
important
today,
right
when
you're,
if
you're
not
in
the
office.
So
you
don't
have
those
water
cooler
moments
in
the
office
to
be
able
to
talk
through
things
or
someone
you
haven't
met
in
those
collisions
now
community
practice.
I
think
a
really
great
way
to
generate
those
collisions
and
sharing
different
ideas
and
perspectives
and
spread
that
knowledge
around.
A
You
know
actually
so
you're
coming
from
fidelity
you're,
also
in
a
highly
regular,
regulated
environment.
Like
me,
you
know,
and
dean
had
brought
a
good
point
which
was
around
you
know
top
down
versus.
You
know
I
guess
for
lack
of
a
bedward
bottom
up.
How
do
you
negotiate
that
difference
between
top
down
and
commuter
practice
model,
because
it
feels
like
there's
a
bit
of
a
there's
a
bit
of
a
disconnect
there
and
how
those
models
can
work?
And
I
I'm
going
to
assume
that
you've
had
to
kind
of
navigate
that
contention.
C
Yeah,
I
think
it's
a
journey
when
we
come
from
traditional
way
of
working
to
new
way
of
working,
so
I
can,
in
my
opinion,
really
the
strength
is,
is
again
going
back
to
why
we
want
this
to
happen.
What's
when
we
think
of
you
know
the
engineering
experience
we
think
of
talent,
we
think
of
development
talent,
we
think
of
how
fast
you
know
our
industries
are
moving.
Now.
Financial
services
was
probably
once
a
slow
moving
fall
down
up
a
hill.
Now.
D
C
Down
the
hill
really
fast,
we
got
to
be
able
to
pivot
really
quickly.
So
many
of
the
issues
that
you
know,
let's
say
pure
technology
teams
had
previously
you
know
financial
service
industries
now
have
to
you
know,
obviously
enable
them.
So
I
think
there's
a
cultural
shift
too,
but
you
know
there's
always
a
balance
in
an
organization,
but
you
know
you
know
when
it
comes
to
in
initiating
that
culture
of
sharing
collaboration,
empowerment
and
again
it's
just
the
guardrails
in
which
you're
working
right
and
I
think
monster
defined.
C
You
know
you
don't
want
to
stifle
innovation
by
putting
too
much
top
down.
You
want
people
to
innovate.
You
want
people
to
be,
you
know
collaborative
across
the
bond,
so
you
know
to
go
back
to
your
question.
You
know
it
that
that's
been
a
journey.
You
know
what
I've
seen
and
working
in
infidelity
in
other
places.
Is
you
know
that
open
and
transparent
you
know
need
to
change
the
culture
and
bringing
a
lot
of
things
we've
talked
about
today
into
the
organization
to
truly
shift
it
and
again?
C
A
Yeah
and
whenever
you
talk
about
culture
change
and
how
and
systems-
and
you
know,
implementing
systems
and
the
organizations
in
which
they're
implemented.
Inevitably
you
come
to
conway's
law
about
how
the
systems
you
implement
end
up
being
a
reflection
of
the
the
culture
from
which
it's
spreading
the
organization
from
which
it
came.
A
And
then
I
guess
the
question
is
like:
how
do
you
use
metrics
to
to
try
to
get
ahead
of
that
or
or
can
you
get
ahead
of
that
and
if
you
have
experiences
from
your
background,
where
you've
seen
metrics
lead
to
that
right,
the
right
kind
of
questions
that
help
to
drive
cultural
change.
I
would
love
to
hear
something
along
those
lines.
C
You
know
so
the
macro
level
allow
us
to
understand.
Are
we
moving
the
dollar?
Are
we
moving
forward,
but
we
still
depending
on
your
context,
where
do
I
start?
I
think
you
mentioned
a
company.
You
were
working
with
understanding.
You
know
the
gamification
of
what
at
the
micro
level
is
you
know
we
when
we
focus
on
the
macro.
C
You
can't
just
leave
them
feeling
around
in
the
dark
right
in
relation
to
how
do
I
change
lead
time
and
particularly,
if
you're
beginning
this
journey
and
the
same,
even
if
you're,
a
very
mature
team,
you're
not
done
because
you're
in
the
cloud
or
you
know,
you're,
building
resilient
systems,
there's
always
room
for
improvement,
but
the
context
is
different
and
how
do
you
guide
teams
and
focus
on
the
best
practices?
The
engineering
excellence?
You
know
part
of
my
role
now
is
the
pursuit
of
mastery
of
software
craft.
A
A
B
Yeah
you
definitely
you
want
to
start
from
the
first
commit.
If
you
can,
we
find
that
the
longer
code
just
stays
in
the
pipeline,
the
more
instability
we
see,
the
more
incidents
we
see.
You
know
your
batches
start
getting
bigger,
which
means
that
it's
harder
to
isolate
and
test
the
the
all
the
changes
that
are
going
into
it
so
and
also
like.
B
We
want
to
consider
these
metrics
they're
predictive
of
you
know
all
these
business
outcomes,
so
we're
talking
about
business
outcomes,
and
so
we
are
considering
that
the
customer
is
waiting
for
us
to
build
these
features
and
push
the
stuff,
and
we
want
to
know
how
quickly
and
how
well
we're
doing
that.
And
thus
we
would
like
to
start
with
our
first
commit.
A
B
F
C
Stage
yeah,
I'm
just
thinking
when,
when
you
think
of
a
metric
like
that,
if
you,
if
you
focus
on
the
at
whatever's,
if
you're
doing
pair
programming
who's,
does
the
commit
right,
you
know,
there's
there's
there's
patterns
of
work
that
you
know
juana's.
But
the
question
is:
what's
the
outcome
you
want
to
achieve
by
measuring
that
and
two
you
know
if
you're
pair
programming
etc.
That
might
be
something
difficult
to
to
even
understand
right.
D
Deployment
per
developer
per
day
was
defined
in
the
accelerate
book
and
so
again
an
average
and
dean.
I
appreciated
you
pointing
out
hey
average
may
not
be
great
median.
Also
may
not
always
be
great
counting.
How
many
days
you're
actually
deploying
maybe
to
production,
is
interesting,
so
we've
been
chasing
some
of
the
deployment
per
developer
per
day,
and
what
we
found
interesting
is
is
corporate
on
boards,
like
agencies
full
of
developers.
That
number
drops
very
quickly.
D
A
B
I
would
say
just
different
metrics
and
I
I
I
would
never
tell
anybody
that
like
a
metric
is
not
useful
if
it's,
if
it's
useful
to
you,
I
I
like
to
think
of
like
there's
almost
like
metric
placebo,
if
it,
if
it,
if
it
drives
real
performance
improvements,
then
I
think
use
it.
But
you
know
back
to
like
when
a
measure
becomes
a
target
ceases
to
be
a
good
measure.
So
I
think
you,
the
nice
thing
about
the
four
key
metrics
is.
B
They
are
they're
predictive
of
success
and
they're
they're
at
this,
like
macro
level,
so
like
if
you're
using
this,
and
you
find
that
it
is
correlating
and
it's
pushing
for
it's
improving
your
deployment
frequency,
then
by
all
means,
keep
using
it.
If
it's
not,
then
maybe
that's
a
time
to
step
back
and
consider
like.
Is
this
a
good
energy?
Is
this
helping
us
we're
not
seeing
the
business
outcomes
that
we
want
to
see.
C
Yeah,
just
to
add
kenny,
maybe
to
the
right
when
I,
when
I
mentioned
about
pair
programming-
and
you
know
what
is
the
effect
of
measuring
it,
you
know,
to
your
point
mean:
is
you
could
have
a
negative
effect?
Maybe
some
of
the
ways
you
want
to
work?
It's
you're,
not
it's
going
to
hinder
you
working
that
way,
because
now
you're
measuring
at
an
individual
level,
rather
than
deployments
to
development
or
qa
or
whatever
right,
which
is
more
a
team
task.
A
I
I
thank
you
guys.
Sorry,
sorry,
thank
you
kitty.
I
thought
he
was
going
to
stay
with
us
anyway.
No,
I
love
this
question
because
it
gets
to
the
heart
of
like
hey.
Do
we
really
need
to
do
this,
which
I'll
be
honest,
a
lot
of
times
when
I'm
presented
with
metrics
projects?
The
first
question
I
ask
is:
do
we
really
need
to
do
this?
A
C
So
you
know
and
that's
where
we
when
we
think
when
I
think
about
the
micro
focusing
on
on
the
practicing
principles,
is
fundamental.
You
want
to
be
doing
the
right
things,
but
which
ones
do
you
focus
on
now
versus
later
and
that's
going
to
depend
on
what
your
constraints
are,
and
you
know
you
know
various
other
inhibitors,
and
you
know,
I
think
the
comment
I
think
is
totally
right.
You
should
do
that,
but
you
know
the
four
keys
allow
you
to
understand.
A
A
Yeah
I
mean
ultimately,
there
has
to
be
some
tie-in
to
core
business
value
right.
Otherwise,
why
are
you
doing
it
and
if
you
can't
demonstrate
that,
then
either
your
process
of
measuring
outcomes
is
wrong
or
the
the
the
the
thing
that
you've
you're
trying
to
do
is
is
not
quite
you
know
in
the
right
direction.
F
F
So
that's
interesting
to
see,
and
we've
got
ten
percent
who
are
doing
three
to
four
well
eleven
point:
five
percent,
which
is
good
and
then
lots
of
interesting
data
as
well
on
the
snapshot
of
how
the
teams
who
are
measuring
are
doing
it.
So
we've
got
internal
scripts,
open
source
projects
and
proprietary
software.
A
Thank
you
now
that
is
an
interesting
poll
result
especially.
Does
your
team
track
door
metrics
an
overwhelming
number
saying?
No,
I'm
curious.
I
wasn't
following
the
chat
script,
but
I
don't
know
if
we
put
a
link
into
the
the
dora
site
or
or
the
four
keys
site
just
to
make
sure
people
have
those
resources
on
hand.
F
F
A
Well,
no,
I
mean
you
could
talk
about
this
for
literally
hours
and
and
maybe
we
should
get
some
sort
of
you
know
informal.
I
don't
know
twitch
stream
or
discord
something
going,
so
we
can
have
a
more
fully
fledged
conversation
with
all
the
other
participants.
That
would
be.
That
would
be
kind
of
fun.
E
A
Thank
you.
Thank
you
nick.
Thank
you
dina.
I
really
appreciate
your
participation
here.
I
think
hopefully
this
has
been
useful
for
those
following
along
and
I
look
forward
to
hearing
from
everybody
in
the
comments
or
maybe
hopefully
we'll
see
you
in
the
slack
channel
or
at
future
end
user
council
sessions.
This
has
been
fun.
Thank
you.