►
From YouTube: Building DevOps Metrics for your Choice of CD Tools Through CD... Andrea Frittoli & Erik Sternerson
Description
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
Building DevOps Metrics for your Choice of CD Tools Through CDEvents - Andrea Frittoli, IBM & Erik Sternerson, doWhile
Continuous Delivery is all about getting your software released, and as software and system complexity continues to increase, so does the need to have many different tools, services and even people involved in building, verifying, deploying and monitoring your software. DevOps metrics help organisations measure their success with continuous delivery however, the plethora of different tools in use make it challenging to collect the data first and to extract coherent metrics from it. In this talk the speakers will introduce the CDEvents project and specification, which establishes a shared terminology and set of CI/CD-related events built on CloudEvents. They will demonstrate how CDEvents helped them gather consistent data from different CD tools, which can be used to calculate DevOps metrics.
A
A
B
A
A
So
this
special
interest
group
works
with
defining
and
establishing
common
terms
in
the
city
space
and
a
lot
of
the
terms
that
end
up
in
the
cd
events
project
come
from
this
interoperability
group.
So
we
we
just
sorry
yeah.
We
benefit
a
lot
from
their
work
from
the
interoperability
group
in
late
2020
spawned
a
work
group
called.
It
was
then
called
events
in
ci
cd.
A
A
With
this
specification,
we
can
then
build
sdks
in
different
languages
for
different
platforms,
and
these
sdks
provide
integrators
and
developers
with
a
nice
experience.
Producing
and
consuming
events
to
just
yeah
make
life
easier,
and
with
these
sdks
we
can
build,
integrations
and
proof
of
concepts.
A
A
A
So
this
again
is
part
of
a
wider
effort.
We
have
the
cd
events
team,
which
me
and
andrea,
are
part
of
so
we're
sort
of
yeah.
I
guess
driving
the
work
to
define
the
specification
and
also
driving
the
work
for
some
of
these
sdks,
but
there
is
a
lot
of
of
really
positive
engagement
with
communities
of
other
projects
like
jenkins
and
other
projects
like
that
who
are
helping
build
sdks
or
even
driving
the
work
for
some
sdks
and
driving.
They
work
for
some
integrations.
A
A
C
A
Presentations
in
previous
conferences,
which
go
into
more
detail
on
what
I'm
going
to
talk
about
next,
so
you
can
go
look
at
those
if
you're
interested
in
more,
but
they
are
interoperability
so
making
things
work
together
and
observability,
so
making
it
easy
for
external
systems
or
people
to
figure
out
what
is
happening
and
how
are
things
going
and
observability
allows
us
to
define
and
calculate
metrics,
and
for
this
talk
specifically,
we
are
looking
at
four
major
devops
metrics.
So
andrea,
would
you
like
to
say
a
few
words
about
those.
B
So
if
you
look
at
the
state
of
devops
report,
which
has
been
published
by
puppet
and
sponsored
by
several
organizations,
I
mean
it's
been
running
for
about
10
years
now.
I
think
and
yeah
they
look
at
how
different
organization
implements
devops
practices
and
they
identified
metrics.
That
can
be
used
as
like
measurable
outcomes
to
qualify.
A
A
So
let's
look
at
a
setup
like
this.
We
have
some
sort
of
orchestrator.
This
could
be
teched
on
controlling
cubectl
to
deploy
stuff.
It
could
be
argo,
spinner,
captain
or
one
of
the
other
devops
automation
tools,
but
their
responsibility
is
to
listen
to
our
requests
and
make
sure
things
are
deployed
that
we
would
like
them
to.
A
So.
Let's
then
say
that
we
get
a
new
delivery
of
our
application,
so
we
get
the
new
version
and
we
also.
Let's
say
we
made
a
configuration
change
to
say
that
we
also
want
to
have
a
new
deployment
of
our
application,
so
one
upgrade
one
new
deployment
in
cd
events.
We
have
messages
for
this
already
for
the
upgrade
we
have
an
event
called
service
upgraded
and
for
the
deployments
we
have
an
event
called
service
deployed.
B
B
B
You
will
have
also
the
change
merged
event
and
the
artifact
published
event
associated
to
that,
and
then
you
will
have
more
changes
than
another
build.
So
in
that
case,
then
it's
enough
to
rely
on
the
repository
and
the
time
step,
but
in
real
life
in
most
use
cases
you
will
have
much
more
complicated
scenarios
to
you
may
have
bills
running
in
parallel,
so
you
may
be
having
multiple
branches
that
you're
working
with,
so
you
may
be.
Backpacking
changes
so
running
a
bill
that
is
actually
on
some
code.
B
B
Unfortunately,
there
is
no
single
common
interface
in
the
different
scm
system
to
ask
this
question,
but
we
hope
that
we
can
foster
this
kind
of
interface
from
being
created
through
this
kind
of
standardization
process
and
of
course,
I
forgot
to
mention
it
in
the
beginning.
But
here
we
are
looking
a
very
simple
use
case,
where
just
one
artifact
that
is
changed,
built
and
finally
deployed,
which
is
the
next
step
and
for
deployment.
Also,
we
have
specific
events
in
this
case,
we're
showing
the
service
deployed
event
and
that
again
you
may
have
multiple
deployment
system.
B
You
could
have
something
like
captain
or
spinnaker
argo
flax,
screwdriver
well,
a
few
of
them.
This
is
not
an
extensive
sorry,
complete
list
anyways
and
the
deploy
event
needs
to
be
correlated
to
the
artifact
publish
event
and
that
can
be
done
through
the
artifact
id.
So
when
deployment
system
knows
that
a
certain
artifact
was
published,
it
uses
that
id
to
retrieve
the
artifact
and
deploying
it
into
the
system,
and
then
this
artifact
id
must
be
included
in
all
the
service
related
events,
so
that
we
may
know
the
specific
artifact
version.
C
B
Of
course,
we
may
have
in
real
life
more
sophisticated
scenarios
where,
instead
of
having
a
single
artifact
being
deployed,
we
have
actually
multiple
artifacts
being
combined
together
and
deployed
into
the
system,
and
so
we
are
looking
into
the
cd
event
specification
at
the
concept
of
compositions.
That
would
help
us
to
identify
and
track
this
kind
of
scenarios
through
events
yeah.
That's
it.
A
So,
with
everything
deployed
next
we
should
ask
ourselves:
will
it
always
work
and
unless
you
are
way
better
developers
and
devops
people
than
I
am,
then
the
answer
is
no,
it
will
not
always
work
so
calculating
the
change.
Failure
rate
is
is
important.
It's
a
good
metric
as
well,
so
we
could
define
this
metric
and
I
think
it
actually
is
defined
as
the
number
of
deployments
over
the
number
of
incidents
that
occur
or
failures
that
occur.
A
D
A
Is
that
there
is
an
error
in
the
application
or
a
bug,
it's
definitely
common
and
definitely
something
that
we
want
to
be
able
to
detect
and
typically
candidates,
but
it
doesn't
have
to
be
just
the
application
itself.
It
can
be
the
configuration
of
the
application,
so
a
user
or
whoever
is
operating
or
deploying
your
application
has
made
an
error
in
the
configuration
that
causes
things
to
either
not
come
up
at
all
or
degrade
in
some
way,
and
it
can
be
completely
unrelated
to
the
application.
A
A
So
we
need
to
find
a
way
to
allow
these
very
different
sources
to
send
incidents
in
a
way
that
we
can
collect
them
all
and
calculate
metrics
from
them.
And
if
you
were
to
ask
me
and
andrea
how
to
do
it,
we
are,
of
course
very,
very
biased
and
we
would
say,
send
an
incident
event.
That's
what
we
want
to
do
so,
given
something
like
this.
We
have
the
two
deployments.
A
A
There
may
absolutely
be
a
case
where
multiple
incidents
events
actually
relate
to
the
same
incidents.
There
is
it's
important
to
send
the
correct
metadata
in
the
incident
to,
for
instance,
understand
that
both
the
application
itself
and
the
monitor
detected
this
incident.
So
it's
not
two
separate
incidents,
but
we
still
want
to
be
able
to
count
it.
So
it's
still
good
if
both
of
them
send
events
and
with
this
information
and
the
number
of
deployments
that
I
showed
you
before,
we
have
enough
information
to
calculate
the
change
failure
rate.
B
B
And
again,
we
have
similar
questions
parallel.
What
can
restore
the
service,
so
it
can
be
that
we
restore
a
service
by
performing
a
rollback
or
maybe
installing
a
brand
new
version
of
our
service,
which
included
a
patch
or
a
fix
for
the
bug
that
was
encountered
could
be
that
we
do
a
configuration
change.
It
could
be
scaling
scaling
up
either
horizontally
or
vertically,
so
adding
more
resources
to
a
service
that
fixes
the
issue.
B
Yeah
so,
but
then,
who
generates
those
those
events
right
about
the
the
resolution
of
the
the
service
being
restored?
So
again
it
can
be
the
orchestrator,
the
sensor
resolution
event
and
a
use
case.
A
specific
use
case
here
could
be,
for
instance,
if
you're
using
canadiaf
to
run
your
service
k
native
is
a
platform
for
doing
serverless
type
of
work.
So
it
means
that
and
canadian
is
an
interesting
feature
that
it's
able
to
it
has
an
auto
scaler
that
can
look.
D
B
B
So
one
thing
that
I
wanted
to
mention
is
that
we
don't
have
all
these
events
defined
fully
spec
yet
in
city
events,
but
we
are
working
into
the
specification
of
these.
There
are
a
few
pieces
of
data
that
we
know
that
are
going
to
be
required
like
the
environment
id,
so
we
need
to
be
able
to
discuss
about
which
environment
a
certain
incident
happened
into.
B
We
need
to
know
which
service
are
we
referring
to
and
we
need
to
know
which
version
of
this
service
are
we
talking
about,
but
other
information
that
we
are
considering
is
probably
going
to
be
optional
because,
as
showing
in
this
feature
that
this
kind
of
event
may
come
from
many
different
parties
and
they
may
have
a
different
kind
of
context
about
the
incident.
So
they
may
know
more
or
less
about.
B
What's
what's
happening,
and
in
fact
one
of
the
things
that
we
are
considering
is
that
for
this
specific
kind
of
events,
we
will
need
to
allow
having
multiple
events
being
sent
about
the
same
incident
resolution
case,
and
then
it
will
be
the
job
of
the
observer.
Finally,
to
kind
of
collate
all
these
events
and
put
all
the
metadata
together
so
that
you
can
use
that
information
finally
to
calculate
the
metric.
B
They
have
a
set
of
events
that
they
use
for
application,
lifecycle
events,
they
call
them
so
they
have
abstractions
about
sli
slos
and
what
they
call
problems.
We
call
them
incident,
so
we
are
still
discussing
about
the
nomenclature
there,
but
they
kindly
offered
to
donate
those
set
of
events
to
the
city
events
projects,
so
we
are
discussing
with
them
and
see
whether
it
would
be
a
good
fit,
and
probably
the
answer
is
yes,
but
we
need
to
kind
of
work
to
align
the
specification
there.
A
So
we're
nearing
the
end
of
the
talk,
and
I
would
like
to
discuss
some
some
key
takeaways
one
is
is
to
calculate
these
metrics.
We
need
a
lot
of
data
from
a
lot
of
different
sources
and
gathering
that
up
requires
or
could
require
a
whole
lot
of
integration
into
your
build
systems,
your
source
code
even
and
orchestration,
and
all
that
stuff,
because
most
people,
typically
don't
have
just
one
single
aligned
environment
and
using
one
solve
everything
platform
to
do
anything,
it's
a
mishmash
of
different
tools
and
services.
A
So
there
are
things
we
can
do
to
help
and
a
common
language
is
one
of
those
things
so,
rather
than
worrying
about
all
these
different
tools
and
services
that
you
need
to
collect
data
from
having
something
like
cd
events
where
you
can,
instead
of
looking
directly
into
all
these
tools,
just
get
standardized.
Events
like
james,
merge,
artifact,
published
service
deployed
and
incident
result
which
we
have
looked
at
today.
A
That
makes
it
a
lot
easier
for
an
observer
to
gather
up
all
the
data
that
it
needs.
I
should
mention
here
as
well
also
the
open
telemetry
project,
which
is
definitely
highly
related
to
parts
of
what
we
are
doing.
They
are
a
bit
more
into
tracing.
We
are
a
bit
more
into
higher
level
or
more
abstract
events,
I
would
say,
but
I
think,
a
collaboration.
There
could
also
really
help
boost
things
forward,
but
me
and
andrea
and
the
rest
of
the
team
are
not
going
to
be
able
to
solve
this
on
our
own.
A
So
as
always,
we
want
to
leave
you
with
a
call
to
action.
There's
a
lot
of
things.
You
could
do
some
very
simple.
That
would
really
benefit
the
cd
events
project,
it's
everything
from
just
providing
feedback
on
the
work
that
we're
doing
or
presenting
your
use
cases.
What
are
you
struggling
with
today
that
could
be
solved
using
events
all
the
way
through
working
with
integrations
in
your
own
projects
or
helping
build
sdks
or
just
getting
involved
in
the
community
so
the
best
way
forward,
for
that
would
be
to
go
to
cdevants.dev
there.
A
B
The
sponsor
showcase,
we
have
a
cd
events
table
with
white
papers.
There
so
feel
free
to
grab
one.
We
have
some
stickers
as
well
yeah
and
if
you
want
to
chat
about
city
events,
yeah
we'll
either
be
there
or
find
us
around
in
the
conferences
not
to
mute
space,
and
we
can
meet
there
at
the
table
and
have
a
chat
yeah.
A
C
A
So
the
way
this
is
going
to
work,
as
I
see
it
as
well
and
you
will
have
to
let
me
know
if
you
disagree
and
react
step
by
step
as
we
start
to
define
the
events
that
one
particular
project
either
needs
to
send
and
by
project
having
something
like
jenkins
or
argo
or
kempton
that
they
either
need
to
send
or
and
or
receive.
Then
they
can
adopt
the
cd
events,
specification
and
sort
of
become
cd
events
compliant.
A
If
you
want
to
call
it
that
so
they
are
sending
and
receiving
events
according
to
the
specification
and
same
would
be-
or
I
think
this
opens
up
an
interesting
venue
for
observability
and
metrics
solutions-
things
that
are
quite
hard
to
do
today,
because
you
have
to
basically
have
to
decide
on
a
subset
of
tools
and
things
you
want
to
support.
But
as
more
things
start
producing
city
events,
you
can
say:
okay,
what
I'm
going
to
focus
on
is
getting
observability
on
things
that
produce
cd
events.
A
So
an
end
user
probably
wouldn't
worry
that
much
about
the
protocol
itself,
but
they
would
benefit
from
using
tools
and
solutions
that
produce
cd
events,
and
that
consumes
the
events.
Of
course,
there
are
end
users
who
may
want
to
build
cd
events
into
their
own,
build
script
or
things
like
that,
and
then
the
way
that
would
approach
it
is
to
just
get
one
of
our
sdks
and
use
that
to
produce
events.
B
Yeah
yeah,
absolutely
I
mean
we
really
hope
to
foster
kind
of
an
ecosystem
of
tools
around
this
city
events.
So
and
then
right.
If
you
are
an
organization-
and
you
have
quite
a
few
of
the
tools
that
may
be
more
listed
below
and
then
you
have
a
new
team
that
certain
skills,
so
they
like,
they
want
to
try
a
new
tool.
As
long
as
these
tools
produce
cd
events,
then
you
can
swap
with
another
tool
or
plug
in
a
new
tool,
and
then
you
still
have
your
metrics
basically
available.
B
E
A
Yes,
so
this
has
been
a
continuous,
I
guess
both
struggle,
but
also
inspirational
for
us
is
to
we're
always
looking
for
like
who
is
the
right
person
to
talk
to
in
these
organizations
to
to
get
a
discussion
started
and,
and
we're
always
trying
to
network
in
one
way
or
another.
So
that's
an
open
call
to
action
to
everyone
as
well.
A
It's
gonna
have
to
be
by
finding
the
right
people
to
talk
to
to
foster
some
enthusiasm
about
cd
events
and
also
help
them
do
the
actual
integration,
and
I
absolutely
agree
having,
for
instance,
github
producing
cd
events
can
really
be
a
good
start
of
a
chain
like
what
andrea
showed
before
in
the
like
change
merged
and
those
kind
of
events.
So,
yes,.
A
D
A
Think
there
will
have
to
be
some
different
strategies
there.
You
could,
I
mean
a
reasonable
assumption,
I
would
say,
is
if
some,
if
two
incidents
relate
to
the
same
deployment
and
they
come
very
close
to
each
other,
it
may
be
good
enough
to
just
say
these
are
probably
the
same
incident.
Even
if
we
can't
prove
that
it's
the
same
incident,
it's
still
yeah,
it
would
affect
the
same
users
the
same
way
anyway.
B
A
So
yes,
unfortunately,
we're
going
to
have
to
wrap
up
because
we
have
another
group
of
people
coming
soon.
But
please
come
and
talk
to
me
in
andrea.
We
will
be
here
at
the
stage
for
a
few
minutes
and
we
will
again
really
like
to
thank
you
all
for
for
showing
up
and
listening
and
cd
events.dev
is
where
you're
going
to
find
us
and
we'll
be
very
happy
to
have
you
involved
one.