►
From YouTube: Measuring DevOps - Dina Graves Portman, Google
Description
Measuring DevOps - Dina Graves Portman, Google
Software metrics often pit different functions against each other and result in local optimizations at the cost of overall outcomes. Google’s research has developed and validated four metrics that provide a high-level systems view of software delivery and performance and predict an organization’s ability to achieve its goals. Accurate collection of the data required to measure the four key metrics can be time consuming, error prone, and incomplete. This session will introduce some automation with Google Cloud Platform that automates the generation and collection of the metrics required to measure the four key software delivery performance metrics.
Slides (PDF): https://static.sched.com/hosted_files/cdcon2020/c0/Measuring%20DevOps%20-%20cdCon%20-%20DinaGravesPortman.pdf
Intro: (0:00)
What's DevOps?: (0:30)
What does DevOps do?: (1:28)
Key metrics: (2:26)
Four Keys Infrastructure: (4:43)
Metrics: (6:28)
How does Tekton fit in?: (10:26)
Demo: (10:42)
Tekton Challenge: (16:24)
Four Keys Dashboard: (18:18)
Try it out! (19:03)
Q&A: (19:19)
A
A
A
Apparently
there
are
a
lot
of
things
that
devops
is
not
it's
a
little
bit
harder
to
say
what
devops
is
lots
of
possible
good
definitions?
Here's
a
good
one
devops
is
an
organizational
and
cultural
movement
that
aims
to
increase
software
delivery,
velocity,
improve
service
reliability
and
build
shared
ownership
among
software
stakeholders
in
a
picture
form.
A
That
is
what
practices
and
capabilities
are
influential
in
driving,
successful
teams,
and
what
they
found
is
that
software
delivery
and
operational
performance
is
key.
If
you
want
to
have
good
business
outcomes,
you
really
need
to
have
good
software
delivery,
performance
and
operational
performance,
so
that
is
to
say,
delivering
software
quickly.
Reliably
and
safely
is
at
the
heart
of
technology,
transformation
and
organizational
performance.
A
And
what
we
find
is
that
people
who
do
well
in
these
areas
they
do
really
really
well.
They
have
both
extremely
strong,
fast
velocity,
velocity
and
extremely
strong
reliability,
and
so
when
we
look
at
elite
performers,
we
see
that
they
are
deploying
code
208
times
more
frequently
than
low
performers.
A
Their
lead
time
from
commit
to
deploy
is
106
times
faster.
Their
change
failure
rate
is
7
times
lower
and
their
time
to
recover
from
incidents
is
2
604
times
faster.
So
our
elite
performers
they're
not
sacrificing
a
velocity
for
stability,
or
vice
versa,
they're
doing
really
well
at
both,
and
so
now
that
we've
talked
a
little
bit
about
what
devops
is
I'm
going
to
talk
about
the
4k
metrics,
which
is
an
open
source
tool
to
collect
and
automate
the
calculation
of
these
four
key
metrics?
A
So
just
to
refresh
they
are
lead
time
to
change
deployment,
frequency
change,
failure
rate
and
time
to
restore
in
the
research.
It
really
focused
on
doing
surveys,
but
we
want
to
do
systems
data
because
we
have
it.
We
actually
have
all
of
the
data
that
we
need
to
calculate
these
metrics
and
if
you
really
want
to
have
an
unbiased
view
of
your
organization,
especially
like
the
trends
and
how
the
performance
changes
over
time,
then
systems
data
is
really
the
way
to
go.
So
that
brings
us
to
our
first
challenge.
A
The
challenge
is
that
developers
use
many
different
kinds
of
tools
and
environments
to
do
their
work.
So
how
do
we
capture
all
of
it,
you're
you're,
doing
deployments
in
one
place,
you're
doing
your
commits
and
your
pull
requests
in
another
place,
you're
filing
your
bugs
somewhere
else.
So
how
do
we
capture
all
of
this
and
how
do
we
account
for
the
unknown
data
sources?
I
perhaps
you
know
I
know
about
git
lab
and
github,
but
maybe
you're
using
bitbucket,
and
so
how
do
I
build
an
open
source
team?
A
The
raw
events
table
is
is
basically
a
devops
log,
so
this
is
where
all
of
that
webhook
data
is
flowing
in
and
it
should
contain
every
event
related
to
development
deployment
and
incident
management.
So
when
in
doubt
you
know
stream
your
data
to
this
table
and
from
the
raw
events
table,
we
create
these
three
derived
dimensional
tables
changes,
deployments
and
incidents,
because
if
we
have
a
view
of
changes
as
an
object
and
a
view
of
a
deployment
as
an
object
in
an
incident,
then
we
can
calculate
the
4k
metrics.
This
is
all
we
need.
A
So
if
you're
using
a
ci
cd
system,
like
you
know,
techton,
you
have
all
the
data
that
you
need
already.
So
if
we
stream
this
information
to
our
table,
we
can
use
it
to
calculate
frequency
and
the
dora
team
bucketed
every
metric
into
elite,
high,
medium
and
low
performing
teams.
So
in
this
case,
daily
deployment
frequency
falls
into
the
high
performing
threshold,
and
so
we
represent
this
with
a
green
badge.
A
Our
next
metric
is
lead
time
to
change,
which
is
defined
as
the
amount
of
time
it
takes
a
commit
to
get
into
production
so
lead
time
to
change
requires
two
important
pieces
of
data
when
the
commit
happened
and
when
the
deployment
happened.
This
means
that
for
every
deployment
we
need
to
maintain
a
list
of
all
the
changes
included
in
the
deployment
which
is
easily
done
if
you're
using
triggers
that
have
a
sha
mapping
back
to
the
commits.
A
With
the
list
of
changes
in
the
deploy
table,
we
can
join
back
to
the
changes
to
you
can
join
back
to
the
changes
table
to
get
the
timestamps
and
then
calculate
the
median
lead
time.
The
dashboard
puts
the
lead
time
into
buckets
defined
by
the
door
research
less
than
a
day
less
than
a
week
less
than
a
month
a
year
and
less
than
a
week
is
medium.
A
Our
next
metric
is
the
change
fail
rate,
which
is
defined
as
the
percentage
of
deployments,
causing
a
failure
in
production
to
measure
the
change
fail
rate.
We
need
to
know
two
things,
how
many
deployments
were
attempted
and
how
many
resulted
in
failures
in
production,
so
our
deployments
are
the
the
denominator
and
the
incidence
are,
is
the
numerator
and
so
to
get
this
number,
we
need
our
total
account
of
deployments
which
we
can
get
from
our
deployments
table
and
we
need
to
link
it
to
incidents.
A
Now.
Incidents
could
come
from
anywhere,
maybe
you
are
automatically
detecting
some
of
them
for
something
like
stackdriver
or
maybe
you
have
a
bug
system
or
maybe
you're,
using
like
labels
on
github
incidents,
or
you
could
have
like
a
form
like
a
spreadsheet
pipeline
or
even
a
fully
fledged
issue
management
system,
which
would
be
ideal.
A
A
Next,
we
have
time
to
restore
this
is
defined
as
how
long
it
takes
an
organization
to
recover
from
a
failure
in
production.
Again
you
need
the
deployments
table
and
the
incidence
table,
but
this
time,
instead
of
just
taking
a
rate,
we
need
to
get
the
timestamp.
So
we
know
the
delta
of
like
when
the
incident
was
created
and
when
it
was
resolved
and
the
incident
will
have
been
created.
You
know
when
the
deployment
happened
that
contained
the
failure
and
then
the
resolution
is
the
deployment
that
fixes
it
similar
to
the
last
metric.
A
A
And
the
first
thing
it
does
is:
ask
us
if
we
wanna
create
a
new
google
cloud
project.
I
say
yes,
I'm
just
going
to
house
all
the
infrastructure
for
the
for
the
four
keys
pipeline
dashboard
and
it's
going
to
ask
us
what
kind
of
version
control
system
and
ci
cd
system
we're
using
I'm
putting
in
github
and
techton,
and
then
it
starts.
Turning
on
all
the
epis,
it
starts.
A
Building
and
deploying
the
the
cloud
run,
workers
it
creates
the
pub
sub
topics
and
subscriptions
includes
the
service
accounts
and
sets
the
permissions,
and
it
also
creates
the
bigquery
tables
and
then
asks
me
if
I
want
to
create
a
separate
new
project
to
test
deployments.
I'm
saying
no,
I
do
want
to
generate
mock
data,
so
I
say
yes
and
then
it
just
it's.
Creating
mock
data
schedules,
the
bigquery
tables,
and
now
it's
going
to
once.
This
is
complete.
A
A
So
gotta
authorize
it
and
then
there's
going
to
be
a
drop
down,
so
you
choose
which
project
you
want
to
use
and
then
connect
and
create
reports.
And
then
you
have
the
option
to
like
not
use
the
template,
but
the
template's
already
set
up.
So
it
pops
up-
and
it
looks
like
this,
and
so
you
can
see
that
we
look
like
we're
a
pretty
decently
performing
team.
A
You
know
got
some
green
badges
and
yellow
badges.
We
have
a
couple
places
where
we
could
improve,
but
all
in
all
this
is
pretty
good,
pretty
good
standing.
So
let's
look
at
the
underlying
data,
four
keys
events
raw
and
we
can
see
that
there's
a
lot
of
data
in
here
all
says,
github
mock.
So
this
is
our
mock
data.
Let's
exclude
it
and
then,
when
we
run
this
query,
we
will
see
that
there's
no
data,
so
we
want
to
integrate
with
the
repo.
A
So
we
can
get
some
data,
real
data,
and
so
first
thing
we
need
to
do
is
we
need
to
get
the
url
for
the
event
handler
and
then
the
second
thing
we
need
to
do
is
we
need
to
get
the
secret.
That's
we're,
storing
it
in
secret
manager,
so
we
can
check
against
it
when
github
send
us
these
events
and
we
want.
We
want
everything.
We
want
all
the
events,
all
the
data
that
happens
there.
We
go
add
webhook
and
there
is
so.
Let's
go
ahead
and
test
this
out.
A
A
A
A
Okay,
so
let
me
show
you
that
it
does
work,
and
so,
if
I
just
commit
here
and
I'm
just
pushing
directly
to
master
so
that
I
can
I
can
trigger
this
when
I
refresh
my
application,
my
hello
world
application
here,
then
it
says
hello,
tech,
town
trigger
so
great,
it's
working,
it's
triggering
it's
passing
all
the
way
through
excellent.
So
now
we
want
to
get
the
data
from
this
kind
of
event
into
our
four
keys
pipeline
and
the
what
I'm
going
to
do
here
is
I'm
going
to
use
the
default
cloud
event
sync.
A
So
this
sends
all
the
cloud
events
that
are
happening,
your
checkdown
pipeline
to
the
url
of
your
choosing,
and
so
I'm
just
going
to
put
an
event
handler.
This
is
a
public
url
though
so
I
do
want
to
like
add
some
kind
of
like
token
or
secret
or
something
so
I'm
just
going
to
put
this
in
kind
of
like
an
api
key
right
here,
so
that
my
event
handler
can
confirm
that
this
is
an
authorized
source,
so
I'm
applying
the
config
there
and
now
that's
updated.
A
So
when
I
update
this
again
and
now,
instead
of
hello
text
instagram,
I
do
hello
four
keys.
Then,
when
I
refresh
oh
yeah,
when
I
look
at
the
data,
I
can
see
all
of
these
techdown
events.
I
can
see
all
of
like
the
task
runs
and
the
pipeline
runs
in
their
different
states,
which
is
really
exciting
and
ultimately,
what
we're
going
to
look
for
is
where
pipeline
run
is
successful
and
that
is
going
to
represent
a
successful
deployment
and
let's
go
ahead
and
look
at
our
app
and
there
it
is.
A
A
So
it's
a
little
difficult
to
create
an
open
source
kind
of
query.
That's
always
going
to
work
for
people,
it's
different
when
you're
working
with
you
know
like
the
github
data
or
like
even
the
cloud,
build
data,
because
those
data
models,
they're
they're
kind
of
set-
and
you
you
know
like
this-
is
a
push
event.
It
says
it
right
there,
and
so
I
know
that
that
is
that
kind
of
change
with
techton
your
your
pipeline
run
doesn't
actually
have
to
deploy
anything
you
could
just
be.
A
A
But
you
know
if
your
definition
of
these
things
is
different.
You
can
edit
it
in
the
sequel,
and
that
is
fine.
That's
why
it's
set
up
that
way,
and
so
here's
the
dashboard.
This
is,
you
know
the
the
end
game
here
in
the
middle
we
have,
the
the
badges
that
are
are
are
the
four
keys
from
the
door.
Research-
and
this
is
you
know,
backed
up
by
six
years
of
research
and
then
around
the
corner
are
the
the
systems
data
over
time,
and
so
this
gives
you.
A
A
A
As
far
as
future
enhancements
go,
definitely
I'm
looking
to
integrate
with
more
and
more
systems,
also
thinking
about
ways
to
create
the
data.
A
little
more
abstractly,
possibly
create
like
it
like
an
api
which
would
help
with
the
integrations
right
now.
Everything
is
relational
databases
and
the
benefit
there
is
that
if
you
have
an
analyst
on
your
team
or
if
you
know
sql,
it's
pretty
straightforward
to
go
in
and
extract
the
information
that
you
need
and
to
say
like
okay.
A
This
task
run
associated
with
it
or
something
like
that
and
so
like
you
can
just
extract
that
from
the
json,
because
the
entire
body
of
the
event
is
is
sent
to
the
table
and
in
its
json
format,
and
you
know
bigquery
and
I
think
most
sql
systems
now
support
json
parsing,
and
so
you
can
use
like
just
like
a
where
filter
and
say
like
okay
on
the
deployment
script,
you
can
say
where
this
equals
this
equals
this.
So
it's
simple
that
way.
A
A
A
The
question
says
we
currently
store
similar
metrics
from
our
jenkins
and
spinnaker
pipelines
and
influx,
and
we
want
to
move
the
data
to
gcp.
Eventually
I
don't.
I
don't
see
why
not,
if
you,
if
you
go
to
the
four
keys
pipeline,
if
you
want
to
just
like
file
an
issue
with
that
request,
I
you
know
I
want
to
support
as
many
things
as
possible
that
people
are
using
so
yeah.
A
If
you
wanted
to
file
an
issue
asking
for
that
feature,
request
that
was
a
great
idea
or
or
you
could,
you
can
build
out
the
integration
and
send
us
a
pull
request,
also
a
great
idea,
because
it's
open
source.
I
think
I
mentioned
that
a
few
times,
oh
yeah,
so
I
see
I
see
the
question
was:
do
you
only
use
tekton
pipelines
whoops?
Do
you
only
use
tekton
pipelines
for
for
cicd?
No,
it
yeah
it
supports
cloud,
build
as
well
and
also
gitlab
ci
cd
and
also
github
ci
cd.
A
A
That's
kind
of
what
I'm
interpreting
that
as
and
if
that's
the
case,
I
would
say
you
can
absolutely
do
that.
If
you
look
at
the
project,
there's
there's
very
little,
I
mean
the
bigquery
part.
It
uses
google
apis,
but
in
terms
of
like
the
actual
like
compute,
it's
using
docker
files,
it's
you
know
using
containers.
So
if
you
wanted
to
to
build
those
and
deploy
them
onto
another
system,
there's
nothing
stopping
you
from
doing
that.
A
A
A
A
A
A
I
think
standards,
data
standards
are
wonderful
and
important.
I
I
think
that
that
is
if
we,
as
a
an
industry
could
agree
on
that
and,
like
you
know,
have
a
schema
in
schema.org
or
something
like
that
that
everybody
would
use.
That
would
be
amazing,
but
things
like
that
take
usually
take
time
to
catch
on
unless
there's
like
a
really
strong
incentive.
A
A
It's
it's
hard
to
incentivize
people
to
do
things
with
the
the
promise
of
rewards
that
come
down
far
down
the
line,
things
like
recipes
or
jobs
in
google
search
when,
like
we
had
the
schema
for
those
people
really
picked
those
up
because
it
affected
how
it
was
represented
in
search.
That
was
like
an
immediate
reward
that
they
could
get
and
they
could
show
up
in
more
search
results.
Something
like
this,
which
is
a
little
more
niche
and
the
benefits
are
not
as
immediate.
You
know
what
what?
What
do
you
get?
A
A
But
I
you
know
we
are
thinking
about
it.
We
are
thinking
about
it,
it
you
know
and
it
it
starts
obviously
at
home
in
your
own
teams.
So
you
know,
starting
with
like
deployments,
what
does
what
does
a
deployment
mean
and
what
pieces
of
data
do
we
need
to
have
represented
in
it?
Obviously
you
need
the
id.
You
need
the
list
of
the
changes
that
were
included
in
it
and
then
you
should
have
things
also
like
traffic,
and
you
know
where
you're
deploying
it
to
the
the
status.