►
Description
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
Keynote: The Four Keys: Measuring Software Delivery - Dave Stanke, Developer Advocate & Roger Martinez, Developer Advocate, Google
Through nearly a decade of research, the DevOps Research and Assessment (DORA) group has identified four key metrics that indicate the performance of a software development team. As leading indicators of the value that software teams provide to their organizations, these metrics can inform best practices and guide continuous improvement. This session describes an easy-to-adopt approach to instrumenting applications and services with DevOps telemetry. Dave Stanke and Roger Martinez will present the Four Keys project, an open source toolkit that automates these measurements and compiles them into an easy-to-use dashboard.
A
Hello,
everyone
thank
you
for
joining
us
today.
We're
going
to
talk
about
the
four
keys,
I'm
dave,
I'm
roger,
and
we
represent
google
cloud
platform.
So
here's
our
tl,
dr
for
today,
first
software
development
can
be
measured.
Yes,
we
can
I'm
going
to
tell
you
how
the
way
we
do
is
with
four
key
metrics
that
capture
the
delivery
of
the
performance
of
your
delivery
team.
How
well
we're
able
to
deliver
software
to
our
business?
A
We
can
collect
those
metrics
using
survey
data
or
using
instrumentation
from
our
systems
and
a
project.
A
research
project
called
dora
reveals
the
key
capabilities
that
predict
success
and
you
can
get
started
by
going
to
g
dot,
co,
slash,
devops
g,
dot,
co,
slash,
devops,
all
right.
What
is
devops
anyway.
There
are
a
lot
of
definitions
and
none
of
them
is
canonical.
A
We
use
this.
One
devops
is
an
organizational
and
cultural
movement
that
aims
to
increase
software
delivery,
velocity,
improve
service
reliability
and
build
shared
ownership
among
software
stakeholders
that
sound.
That
sounds
cool
right.
Raise
your
hand
if
you
think
that
sounds
like
maybe
kind
of
cool.
Okay.
A
How
raise
your
hand
if
you
know
exactly
how
all
right
we're
working
on
that
and
to
figure
that
out
to
answer
that
question?
There's
a
research
group
called
dora,
that's
devops,
research
and
assessment.
This
group
started
almost
10
years
ago
independently
and
then
became
part
of
google
cloud
in
2018.
It
was
founded
by
dr
nicole
forsgren,
as
well
as
colleagues
like
jazz,
humble
now.
This
is
a
group
of
scientists
that
really
look
at
how
software
teams
work
or
do
they
work.
What
makes
them
tick?
A
What
makes
some
work
better
than
others,
and
I'm
going
to
show
you
a
little
bit
about
the
research
methods
that
we
use.
First
off.
We
gather
data
lots
of
it,
so
we
conduct
an
annual
survey
across
tens
of
thousands
of
professionals
to
look
at
what
things
do
they
do
and
the
the
group
of
people
that
we
look
at
is
really
very
broad,
so
international
right,
we
got
people
from
all
over
the
world,
it's
a
little
biased
towards
north
america
and
europe,
but
we,
I
think
together.
A
Over
half
of
respondents
have
more
than
10
years
of
tech
experience
with
the
the
role
that
they
have
varying
from
an
ic
to
a
senior
manager
and
the
company
sizes
go
from
very
small
to
very
large
and
we
send
them
a
survey
and
it
takes
about
20
minutes
to
complete
and
they
do
it
and
in
the
process
they
learn
a
little
about
themselves.
It's
actually
what
we
hear
from
a
lot
of
people
is
that
answering
the
questions
in
our
survey
takes
the
time
to
pause
and
say
what
am
I
doing?
A
What's
my
deal,
if
you
want
to
have
that
experience
of
taking
that
reflective
pause,
I
encourage
you
to
check
out
that
link,
bitly,
slash
soda2022,
you
can
grab
it
at
the
qr
code.
You
can
participate
right
now.
The
survey
is
happening
right
now.
I
want
your
data,
not
in
a
creepy
way
your
anonymized
data
all
right
now.
What
do
we
do
with
that?
We
apply
rigorous
statistical
methods
to
create
a
model,
a
predictive
model,
about
what
things
happen.
What
happens
after
you
do
those
things
it
creates
a
model
that
looks
like
this.
A
A
You
can
find
the
state
of
devops
report
at
bitly,
slash
dora
soder,
it's
free,
it's
a
pdf,
it's
meant
for
consumption
by
executive,
so
it's
got
lots
of
cool
colorful
charts
and
even
if
you've
read
this
years-
or
this
is
last
year,
of
course,
even
if
you
read
that
go
back
into
the
archives,
because
there's
deep
dives
every
year
on
different
topics,
it
all
kind
of
adds
up
to
this
model.
Now
we
like
to
read
this
model
from
right
to
left.
A
We
start
with
what
are
we
trying
to
achieve
we're
trying
to
achieve
commercial
outcomes?
What
does
that
mean?
That
means
the
things
that
our
board
cares
about,
the
things
that
our
ceo
cares
about
right,
the
bottom
line,
so
profitability
or
market
share,
or
even
things
like
stock
price
right.
Those
are
our
outcomes
from
this
entire
model.
Now
they
are
predicted
by
software
delivery
performance
and
that's
a
key
word.
There
predict
it
doesn't
mean
cause
because
it's
not
deterministic,
it
doesn't
mean
correlate,
because
it's
more
than
just
a
coincidence.
A
It
is
a
predictive
analysis,
the
more
we
do,
the
things
towards
the
left,
the
more
likely
we
are
to
get
the
things
on
the
right
software
delivery
performance
can
be
measured,
hold
that
thought,
but
it's
predicted
by
a
number
of
capabilities.
Now
these
capabilities
are
the
things
we
do
in
our
team
and
we
have
a
catalog
of
them
that
you
can
find
at
g
dot,
co,
slash
devops,
and
this
includes
technical
capabilities
like
trunk
based
development
or
shifting
left
on
security.
A
Now
we've
tried
to
answer
this
question
for
a
long
time.
Has
anyone
here
ever
been
compensated
based
on
lines
of
code
written?
Did
anyone
ever
write?
A
cool
script
that
would
just
generate
a
whole
bunch
of
useless
lines
of
code.
Well
done,
you
did
your
job
really
well,
according
to
that
machine.
What
about
in
terms
of
defects,
fixed
right,
there's
a
great
dilbert
cartoon
about
how
someone
says
you're
going
to
write
himself
a
minivan.
A
These
things
are
deeply
flawed
and
engineers
are
real
smart
about
how
to
take
advantage
of
those
flawed.
That
I
think,
has
caused,
in
my
mind
at
least
a
little
bit
of
skepticism.
Can
we
even
measure
this,
but
through
our
research
we
have
found
reliably
that
we
can
measure
it
and
that
the
measurements
that
we've
used
are
really
predictive
of
those
outcomes
we're
looking
for,
and
it
boils
down
to
these
four
key
metrics
two
for
speed
two
for
stability,
we're
going
to
look
at
them
one
by
one
deployment
frequency.
A
How
often
do
we
release
code
to
users,
so
the
user
can
see
it?
Is
that
a
couple
times
a
year
a
couple
times
a
month
couple
times
a
week
multiple
times
per
day,
the
devops
movement
was
in
large
part,
started
by
the
talk
by
flickr
that
said,
10
deploys
per
day
and
that
blew
people's
minds.
It's
still
a
place
that
a
lot
of
teams
aren't
there
yet,
but
we're
here
to
help
you
move
faster,
get
closer
to
that
ideal.
A
The
other
speed
metric
is
the
lead
time
for
changes.
How
long
does
it
take
from
when
we
commit
code
into
our
version
control
to
when
it's
available
on
prod
for
our
users
to
take
advantage
of
it
right?
This
is
sort
of
the
opportunity
cost
time
how
long
until
from
when
we
want
to
try
and
get
some
code
out.
How
long
does
it
sit
in
our
pipelines?
Kind
of
rotting
away
before
users
can
actually
make
something
of
it
now
going
fast
is
great,
but
we
also
want
to
not
break
things.
Sometimes
that
can
be
attention.
A
Often
it's
not,
and
what
we
found
through
a
lot
of
our
research
is
that
these
things
actually
go
together.
Change
fill
rate
says
when
we
deploy
things.
How
often
does
it
blow
up
in
our
faces,
and
we
say
oopsies
or
perhaps
saltier
language?
I
know
that
my
change
fell
right
back
when
I
was
running
a
service
was
50,
because
someone
said
to
me,
you
know
dave.
It
seems
like
every
time
you
do
a
deploy.
A
Sometimes,
when
we
compare
using
a
cluster
analysis,
our
low
performers
versus
our
high
performance
from
our
research,
we
find
that
high
performers
or
elite
performers
actually
have
thousands
of
times
faster,
lead
times,
thousands
of
times
faster
recovery
from
incidents
at
the
same
time,
their
deploy
their
change
failure
rate
is
three
times
lower
and
they're
getting
code
to
their
users
a
thousand
times
faster.
Now,
if
you
want
to
find
out,
how
are
you
doing?
Are
you
a
low
medium
high?
You
can
take
a
self-assessment
and
you
can
go.
A
Do
this
right
now
takes
about
five
minutes:
g
dot,
co,
slash
devops
your
favorite
website
this!
This
this
test
will
apply
our
statistical
model
to
your
quick
answers
and
you
can
find
out
based
on
your
responses,
whether
you
are
low,
medium,
high
or
elite,
whichever
category
you're
in
good
for
you.
What
matters
is
that
we're
working
to
figure
out
how
to
improve,
and
this
framework
can
help
you
now.
B
B
So
the
quick
check
assessment
that
dave
was
just
talking
about.
It
really
does
take
literally
a
couple
minutes.
It's
like
a
couple
of
questions.
It's
a
great
place
to
start,
so
the
results
are
a
great
indicator
of
what
opportunities
exist,
to
improve
your
processes
and
hopefully
move
you
towards
an
elite
rating
and
even
if
you're,
an
organization
that
has
an
elite
rating
that
won't
really
last
forever.
So
how
do
you
make
sure
that
you're
maintaining
that
rating,
or
at
least
improving
it
so
as
valuable
as
a
quick
check
survey?
B
B
Is
it
configure
some
resources
on
google
cloud
platform
or
on
google
cloud
that
together
take
in
different
events
from
different
services
like
github,
git,
lab
and
others
via
web
hooks
and
then
ingest
that
data
into
an
analytical
database
bigquery,
so
that
data
in
bigquery
is
then
used
to
create
dashboards,
and
that
lets
you
track
and
report
on
your
lead
time,
deployment
frequency
time
restore
and
change
failure
rate,
which
I'm
going
to
say
a
few
more
times.
So
we
can
all
remember
those
four
key
metrics.
B
So
the
cool
thing
about
this
versus
a
survey
is
that
it
uses
actual
factual
data
from
your
pipelines
and
that
lets
you
make
changes
and
adjust
your
processes
and
procedures,
while
continuously
tracking
the
four
keys.
So
that
way,
you
can
identify
patterns
and
trends
over
time,
and
that
increases
your
agility
by
responding
to
those
trends.
A
lot
quicker.
B
So
here
are
some
examples,
because
simply
tracking
the
four
keys
has
been
a
pretty
big
game:
changer
for
a
lot
of
big
organizations
and
sometimes
it's
smaller
teams
within
those
organizations,
but
you'll
see
how
the
the
the
steps
that
those
small
teams
make
really
impact
organization
as
a
whole
and
that's
regardless
of
the
dora
rating.
By
the
way
this
can
be.
You
could
take
the
the
survey
now
and
you
come
out
with
low
medium
high
or
elite,
and
there's
still
always
going
to
be
opportunities
for
you
to
make
improvements
to
your
processes
and
pipelines.
B
B
One
team,
however:
they
discovered
that
their
lead
time
for
changes
was
trending
up.
So
if
nothing
was
done,
they
knew
that
they
could
easily
lose
that
elite
score,
so
it
might
sound
like
they
discovered
a
problem
like.
Oh,
no
we're
going
to
lose
that
score.
Our
lead
time
is
going
up,
but
that's
not
true,
because
nothing
lasts
forever,
like
I
said
before,
what
they
actually
found
was
an
opportunity.
B
So,
after
that
they
had
discussions
between
different
teams
and
they
investigated
the
possible
causes
for
the
increasing
lead
time
and
what
they
found
was
that,
for
example,
some
production
deployments
for
new
releases.
They
were
being
held
up
by
days
or
even
weeks.
They
found
that
they
realized
that
the
work
was
broken.
The
work
that
they
were
doing
was
being
broken
down
too
granularly
and
that
led
to
a
lot
of
the
unnecessary
dependencies
and
that
inevitably
led
to
more
delays.
B
B
So,
based
on
everything
that
they
learned,
they
were
able
to
implement
a
lot
of
impactful
changes
across
the
organization
and
they
reversed
that
upward
trending
lead
time.
So
that
was
just
by
tracking
those
metrics.
That
was
an
elite
rated
organization
who
got
the
best
score
possible
on
that
door
assessment.
B
They
were
able
to
reverse
that
trend
and
kick
off
a
process
of
discovery
and
collaboration,
and
that
led
them
to
identify
opportunities
and
then
that
led
them
to
implement
those
changes.
So
it
didn't
just
improve
that
lead
time
and
retain
that
status,
but
it
also
it
helped
them
better
serve
the
business
seeds
of
their
organization.
B
Here's
another
example-
and
this
is
opposite,
so
the
other
team
in
liberty,
mutual-
they
had
an
elite
score.
This
is
a
brand
new
young
scrappy
team
at
ford
credit
company,
which
is
the
financial
services
arm
of
the
ford
motor
company,
so
that
particular
team.
To
give
you
a
little
bit
more
context.
They
handle
the
lease
and
payment
services
that
ford
motors
offers
to
consumers.
So
one
of
those
services
is
state
to
state
lease
transfers
and
as
a
customer,
if
you're
moving
from
one
state
to
another
which
I've
done
total
nightmare.
B
B
So
the
personal
lifetime
communications
team,
which
is
responsible
for
that
system
for
state
to
state
lease
transfers.
They
got
wind
of
the
dora
research
and
the
state
of
devops
report
and
they
became
interested
in
finding
out
how
they
fare
with
their
lead
time,
deployment,
frequency
time,
restore
and
change
failure
rate.
So
in
the
summer
of
2021,
they
stumbled
on
the
four
keys
project
and
they
started
measuring
those
metrics.
B
So
once
again,
they
saw
a
huge
opportunity,
a
few.
In
a
few
months,
within
a
few
months,
they
were
able
to
make
changes
to
their
process
using
pipelines
they
switched
over
to
trunk
based
development
models.
They
made
the
big
changes
that
they
were
making
to
their
code
base
became
smaller
batches
of
work.
B
B
So
over
time,
naturally,
they
saw
their
their
key
metrics
improve
and
another
number
that
improved
was
that
that
number
of
customer
service
calls
for
state
to
state
lease
transfers.
So
there
are
less
customers
having
issues
and
the
teams
that
handle
support
tickets.
They
had
a
lighter
load
because
of
that
and
as
they
collected
feedback
about
their
systems,
they
were
eventually
able
to
incorporate
those
improvements
to
their
system
and
redeploy
changes
within
two
days.
B
A
Well
before
we
decide
what
to
do
now,
it's
important.
We
all
ask
why
right?
Why
should
we
do
it,
and
I
want
to
make
the
point
that
these
numbers
are
really
cool,
really
impressive.
We
showed
5's
with
a
lot
of
thousands
and
30
goes
to
4
500
great,
but
numbers
are
just
numbers,
so
it's
really
important
that
we
put
this
in
the
context
of
why.
Why
do
we
care
about
making
our
software
delivery
more
efficient
and
and
more
stable
and
the
reasons
got
to
be
about
a
human
being
somewhere?
A
So
a
customer,
a
stakeholder,
a
even
just
a
partner
team
and
something
we
heard
from
all
these
success
stories-
was
that,
through
the
work
of
doing
the
iterative
process
of
improving
on
their
devops
performance,
they
were
able
to
release
better
product
release
safer
a
lot
of
times
this
stuff
can
really
it
can
lead
to
people
sleeping
better
at
night.
Judge
humble
who
I
mentioned
was
one
of
the
founders
of
the
research
says
that
continuous
delivery.
He
literally
wrote
the
book
on
continuous
delivery.
A
He
says
continuous
delivery
is
the
ability
to
deliver
changes
to
production
safely
quickly
and
reliably.
I
think
at
any
time
of
day,
basically,
it
means
making
delivering
software
our
day,
job
right
and
what
we've
seen
through
our
research
is
that
a
lot
of
these
capabilities
that
we've
shown
you
are
predictive
of
achieving
that
continuous
delivery.
So
do
this,
because
it's
good
for
your
users
do
it,
because
it's
good
for
your
own
sanity
and
use
this
framework
to
find
where
you
can
improve
any
improvement
made
anywhere
other
than
the
bottleneck
is
an
illusion.
A
A
It's
the
open
source
project
you'll
also
find
information
about,
like
our
slack
channel,
where,
which
is
where
we
happen,
to
meet
roger
from
your
roger
meet,
will
from
from
ford,
also
take
the
survey,
contribute
to
this
year's
research,
and
you
know,
chat
with
us,
we'll
be
around
find
us
on
slack,
find
us
on
twitter
or
find
us
right
here
in
the
room.
We'd
love
to
speak
to
you
about
what
you're
doing
and
how
you're
going
to
improve.
Thank
you.