►
From YouTube: Sponsored Session: How to “SRE” your Way to Pipeline Improvement - Anders Wallgren, CloudBees
Description
Sponsored Session: How to “SRE” your Way to Pipeline Improvement - Anders Wallgren, CloudBees
Oftentimes, IT professionals think auditability and observability are separate concepts, when in fact, they’re two sides of the same coin. It’s the review of what happened and why - just from different perspectives and moments in time. The point is, building an auditable pipeline should also result in building an observable pipeline.
Most IT pros, however, think about observability in regards to the observability of an operational system. But what if we considered the observability of an internal organizational system instead? For example, the CI cycle time of your pipeline.
To test this hypothesis, Site Reliability Engineering (SRE) approaches can be applied to pinpoint what metrics we care about most and how to align context with desired outcomes.
During my talk, I will share how to place service level objectives (SLO’s), indicators (SLIs), and agreements (SLA’s) into different parts of your Software Delivery Lifecycle. You will learn how to make your pipeline metrics adjacent to auditability and observability standards as part of any continuous improvement process.
What Attendees Will Learn:
How to put SLIs, SLAs, and SLOs in place to align with key pipeline metrics
Examples of how to “SRE” your way to better cycle times (as an example)
How to avoid “metricing yourself to death" in the process
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
A
I've
had
the
great
fortune
of
working
with
our
customers
and
our
engineering
teams
for
many
many
years
working
on
how
to
improve
optimize,
ci
cd
pipelines
anywhere
from
you
know,
basically
commit
to
customer
so
not
necessarily
focusing
today
on
on
just
end-to-end
pipelines.
You
know
some
a
lot
of
this
will
apply
to
whether
you're
doing
a
continuous
integration
pipeline,
continuous
testing
pipelines,
continuous
delivery
pipelines.
Really,
you
know
kind
of
across
the
board.
This
will
hopefully
be
useful
to
you.
So
I've
broken
this
down
into
into
a
few
sections.
A
The
first
one,
I'm
just
gonna,
do
a
few
slides
talking
about
metrics
and
how
to
use
them
on
how
to
use
them.
How
to
drive
outcomes
that
that
sort
of
thing
talk
a
little
bit
about
the
very
small
subset
of
sre
that
I'm
going
to
apply
to
to
improving
pipeline
outcomes
and
then
also
a
little
bit
on
really
what
we're.
A
So,
let's
dive
right
in
and
and
talk
a
little
bit
about,
metrics.
First
metrics
really
are
the
art
and
science
of
continuous
improvement,
really
at
the
heart
of
that
right
and
and
why
metrics
well
well,
science,
because
science
observation
is
really
fundamental
to
the
scientific
method.
You
know
what
we
all
want
to
engage
in
here
is
you
know
we
have
an
observation,
we
kind
of
have
a
question
about
that.
We
form
a
hypothesis
about
why
that
might
be.
A
We
repeat,
you
know
lather,
rinse
and
repeat,
and
this
is
continuous
improvement
and
we
so
we
want
to
use
metrics
as
as
part
of
that,
and
you
know
what
does
that
have
to
do
with
devops,
you
might
say
well,
I
would
argue,
and
lots
of
people
would
argue
that
devops
really
uses
the
scientific
method
as
part
of
enabling
software,
innovation
and
and
that's
kind
of
core
right,
we're
always
running
experiments,
and
I
think
you
can
argue
that
something
like
ci
is
an
experiment.
Will
my
code
changes
work?
Cd
is
an
experiment.
A
Ct
continuous
testing
is
an
experiment
right
and
what
we're
looking
for
is
we're
looking
for
feedback
on
on
how
successfully
we're
doing
those
activities
and
when
it,
when
it
comes
to
metrics.
What's
what's
really
important
is
that
you
choose
metrics
that
drive
your
desired
outcomes
right,
where
there's
lots
of
reasons
to
use
metrics
right,
we
might
want
to
try
to
make
ourselves
look
good.
We
might
want
to
make
ourselves
look
bad.
A
You
know,
I
think,
we've
all
sat
in
those
meetings
or
walked
into
those
kinds
of
you
know
boardrooms
and
conference
rooms,
where
you
know,
there's
there's
tons
of
monitors
or
tons
of
displays
showing
lots
of
metrics,
and
you
know
it
looks
like
norad
or
nasa
mission
control
right
that
that's
not
really
what
we're
after
here.
What
we're
after
here
is.
A
We
we
really
just
want
to
focus
in
on
you
know:
small
is
better
small,
is
better
here,
just
a
few
metrics
that
are
going
to
drive
outcomes
that
that
we
want
to
achieve
in
our
in
our
pipelines,
switching
gears
a
little
bit
and
and
I'm
going
to
just
quickly
talk
about
a
very
tiny
subset
of
google's
sre
topic.
If
you
will
site
reliability,
engineering,
the
the
google
sre
book
is
it's
a
great
read
it's
free.
A
There
are
are
three
things:
service,
level,
indicators,
service
level,
objectives
and
service
level
agreements,
or
I
I
sometimes
will
call
them,
especially
in
this
context,
service
level,
actions
and
and
really
what
we're
talking
about
here
is,
as
the
slide
says,
is
a
service
level
indicator.
Is
it's
a
metric
right?
It's
something
that
you
measure,
and
you
know
in
our
example.
A
Here
the
example
kind
of
builds
out
a
complete
sentence
if
you
will-
and
so
the
the
first
part
of
the
sentence
here,
is
that
the
the
the
indicator
that
we're
that
we're
interested
in
is
customers
can
log
in
and
view
their
data
right
now
around
that
indicator.
We
want
to
derive
an
objective
and
really
that's
a
target
value
for
this
indicator
right.
It's.
It
represents
an
ongoing
commitment
that
we
want
to
keep
this
indicator
in
a
particular
range
or
above
or
below
a
certain
number.
A
So
in
this
case
you
know
kind
of
continuing
to
build
our
sentence
here,
99
of
the
time,
so
customers
can
log
in
and
view
their
data
99
of
the
time
and
then
the
last
part
of
it
are
service
level
agreements
or
or
actions.
If
you
will,
and
that
really
is
a
definition
of
what
happens
when
we
don't
meet
the
objective
for
a
particular
indicator
right,
so
customers
can
log
in
and
view
their
data
99.9
of
the
time
or
they
can
request
a
refund
for
losses
incurred
due
to
unavailability
of
of
the
service.
A
A
For
me
what
the
point
of
of
the
whole
entire
ci
cd
pipeline
process
is,
and
and
and
the
fundamental
most
important
aspect
of
a
ci
cd
pipeline
is
that
it
represents
for
us
a
stable
quality
signal
right.
It's
something
against
which
we
can
test
changes,
changes
to
our
code,
changes
to
our
environment,
changes
to
our
tests
and
expect
to
get
back
a
reliable
indicator
of
whether
we
broke
something
whether
something
is
working
or
not,
working,
how
well
it's
working
or
not,
working
and
and
the
stability
of
that
signal
is
really
really
fundamentally
important.
A
In
fact,
you
know
when,
when
we
come
to
talk
about
the
outcomes
that
we're
trying
to
achieve
here
with
our
with
our
little
experiment
here
our
thought
experiment-
I
I
would
argue-
and
I
and
I-
and
I
think
I
can
argue
very
strenuously
and
strongly-
that
the
stability
of
this
quality
signal
is
in
fact
more
important
than
the
speed
and
performance
of
that,
and
just
to
be
more
concrete
about
it.
The
stability
and
reliability
of
our
pipelines
is
more
important
than
the
speed
of
our
pipelines.
Now
absolutely
we
want
fast
pipelines
right.
A
Not
saying
fast,
isn't
important
fast
is
very
important,
but
stability
is,
is,
in
my
mind,
even
important,
and
so
that's,
if
you
take
anything
away
from
this,
you
know
one
of
the
things
I'd
like
you
to
take
away
is
that
the
stability
of
our
pipelines
is
of
paramount
importance.
So,
given
that
I've,
I've
just
picked
here
just
to
keep
it
simple,
two
outcomes
that
we
want
to
drive
out
of
our
out
of
our
kind
of
thought
experiment
here.
The
first
one
is
that
our
ci
cd
pipeline
should
provide
a
stable
quality
signal
right.
A
A
We
don't
have
a
stable
environment
to
operate
in
against
the
which
to
test
our
changes
and-
and
it's
key
to
have
that
stability
right,
because
otherwise
we
don't
know
whether
the
problem
is
in
the
change
that
we
introduced
or
inherent
or
already
present
in
our
environment
or
in
our
tests
or
or
or
in
our
pipeline
in
our
processes,
so
that
that
that's
very
important
and
I'll
I'll
drive
on
that
point
quite
a
few
more
more
times.
But
but
our
number
one
outcome
is,
we
want
to
provide
a
stable
quality
signal
here.
A
A
Then
you
know
your
your
objectives
are
going
to
be
different
if
you're
operating
in
what
is
for
you
today,
a
flaky
environment
where
things
are
not
very
stable,
then
the
kinds
of
the
kinds
of
objectives
you're
going
to
drive
for
are
going
to
be
a
little
bit
different
than
if
today
you
have
a
very
stable
environment,
and
you
want
to
keep
that
way.
So
some
I'm
sort
of
going
to
be
purposefully
ambiguous
about
those,
though
I
will
talk
to
them
of,
of
course.
A
But
you
know
our
goal
here:
is
you
know
kind
of
in
a
jokey
way
here?
You
know
we
want
zero
days
without
a
ci
cd
failure
right.
We
we
want
a
job
site.
That's
you
know,
humming
along
very,
very
well
so
sli
service
level
indicators
that
we
can
use
to
drive
ci
outcomes
so
on
under
the
the
outcome
of
stable
quality
signal.
I've
chosen
three
here,
so
we're
gonna.
We're
gonna
want
to
keep
track
of
the
success
and
failure
ratio
of
our
pipelines.
A
We're
gonna,
keep
track
of
the
mean
time
to
fix
a
pipeline
when
it's
broken
and
then
we're
also
going
to
keep
track
of
the
root
cause
of
these
pipeline
failures,
and
that's
really
important,
because
the
the
the
what
we
do,
the
actions
that
we
take
are
going
to
be
somewhat
dependent
on.
Why
something
failed
right.
So
we
do
want
to
keep
track
of
that
on
the
fast
feedback
and
fail
fail
fast
kind
of
outcome.
What
we're
gonna
look
for
is
a
couple
of
things.
A
We're
gonna
look
at
our
mean
elapsed,
time
for
a
successful
ci
cd
pipeline
run,
and
I
very
specifically
hear
say
a
successful
cicd
pipeline.
I'm
not
interested.
I
don't
want
to
pollute
the
data
here
with
how
long
a
failed
pipeline
takes
to
run
because
a
failed
pipeline
can
take
a
lot
longer
or
a
lot
shorter
to
run
than
a
successful
one
right.
A
If
you've
got
tests,
timing
out
and
those
sorts
of
things
then
your
pipeline
may
take
you
know
two
or
three
or
five
or
20
times
as
long
to
run
as
when
it's
successful
or
if
it
fails
fast.
It's
going
to
be
very
short,
but
what
we're
concerned
about
here
is
what
is
the
elapsed
time
for
a
successful,
ci
cd
pipeline
and
then.
A
Secondly,
the
second
indicator
on
the
fast
feedback
fail
fail
fast
outcome
that
we're
looking
for
is
what
is
the
mean
time
to
failure
from
the
start
of
our
ci
cd
pipeline
now
I'll
get
into
a
little
bit?
Why?
I
think
that's
a
useful
indicator
to
use
and
it
really
has
to
do
with
why
something
failed,
which
ties
a
little
bit
to
the
root
cause
and
then
what
we
can
do
to
improve
upon
that
and
that
gets
into
was
it
a
system
test,
an
integration
test,
a
unit
test,
an
environment,
a
deployment.
A
You
know,
though,
those
sorts
of
things
so
then
I've
I've
broken
the
service
level
objectives
into
one
slide,
one
for
each
of
the
outcomes
that
we
want
to
that.
We
want
to
drive
and
under
under
the
outcome
of
stable
quality
signal
under
the
the
success
failure
ratio
indicator
really
here.
What
we
want
to
do,
ideally,
of
course,
is
drive
to
zero,
but
be
realistic
about
your
current
state.
A
Right
I
mean
if
today
you
know
one
out
of
five
or
one
out
of
ten
or
one
out
of
two
of
your
pipelines
failed,
and
you
know
you
use
that
as
a
baseline
and
just
you
know,
set
a
goal
to
improve
that
and
and
really
drive
towards
that.
You
know
drive
it
to
zero,
that's
where
we
want
to
get
to,
but
but
you
know
it
doesn't
necessarily
do
any
good
to
just
say,
drive
to
zero
immediately.
A
You
know
you,
you
may
want
to
stage
your
way
in
there
and,
of
course,
keep
in
mind
that
the
longer
your
cycle
times
are
the
more
painful
these
failures
are,
so
we
really
do
want
to
keep
it
as
a
ratio
right,
but
the
absolute
numbers
are
not
that
interesting.
10
failures
of
the
pipeline
that
runs
for
15
minutes
is
very
different
than
10
failures
of
a
pipeline
that
runs
for
15
hours,
service
level.
Objectives
around
mean
time
to
fix.
A
A
If
you
can't
fix
something
quickly,
revert
revert
that
commit
pull
it
out
right,
that's
oftentimes
the
best
way
to
get
the
pipeline
back
to
green,
and
then
you
you
proceed
in
parallel
to
figure
out
why
that
commit
caused
a
problem
right
again
here.
Your
cycle
time
is
good,
is
going
to
impact
this.
You
know
positively
and
negatively
in
the
sense
that,
if
you've
got
a
long
cycle
time
and
that
that
pipeline,
that
failed
is
still
running,
you
know
you
may
have
subsequent
failures
to
to
discover.
So
you
may
want
to
keep
it
running.
A
That
does
give
you
a
little
bit
of
air
cover
a
little
bit
of
time
to
figure
out.
Why
why
something
failed?
On
the
other
hand,
if
if
it's
a
persistent
failure
in
a
rapidly
running
pipeline,
you're
pretty
quickly
going
to
have
a
people,
you
know
set
of
people
lined
up
outside
your
door,
either
physically
or
or
virtually
wanting
you
to
fix
this
and-
and
you
know
mean
time
to
discovery-
is-
is
very
related,
obviously
here
to
meantime
to
fail.
A
So
so
that's
you
know,
that's
related
to
this,
but
I
really
what
I'm
concerned
about
here
is
mean
time
to
fix
from
the
time
that
we
discovered
that
that
we
have
a
problem
and
and
other
things
that
you
want
to
think
about
in
terms
of
what
kind
of
objectives
you
want
to
throw
around
this
is:
do
we
have
all
committers
involved?
Are
they
all
in
the
same
time
zone?
Are
we
all
in
the
same
office?
Are
we
spread
across
the
globe
right
that
can
impact
your
objectives,
your
impacts?
A
So
you
might
want
to
tighten
up
your
objectives
and
then
the
main
reason
why
I
pulled
the
root
cause
indicator
in
here
is
really
for
the
first
two
right,
because
they
are
so
important
to
the
outcome
of
of
of
having
a
a
stable
quality
signal
here
and
that's
flaky
tests
and
flaky
infrastructure
right
these.
You
really
want
to
drive
to
zero
and
and
if
you
suffer
from
flaky
testosterone,
you
really
probably
ought
to
go
down
the
road
that
I
described
before
around
run,
10
or
50
or
100.
A
A
So
that's
you
know,
there's
a
little
bit
of
ambiguity
there,
but
but
start
by
running
ten
of
them
just
run
ten
of
them
and
see
what
happens
right,
collect
the
root
causes
then
go
track
down
the
flaky
tests
track
down
the
environment,
problems
or
or
what
have
you
and
then
the
the
other
kinds
of
failures
are
going
to
be
more
sort
of
the.
You
know
kind
of
human
mistakes
right
I
have
a
bad
commit.
I
caused
a
compiler
failure.
A
I
forgot
to
add
one
of
the
files
to
my
or
stage
one
of
my
files
into
the
commit.
So
get
didn't
pick
it
up.
You
know
we're
human.
We
make
mistakes.
The
whole
point
of
ci
is
to
catch
these
mistakes.
The
whole
point
of
continuous
delivery
pipelines
is
to
catch
these
mistakes
before
the
customer
benefits
from
them.
So
you
know
if,
if
we,
if
we
had
100
success,
you
know
maybe
we're
not
trying
hard
enough
to
find
failure.
You
know
quite
frankly,
but
you
know
it
could
be
a
bad
merge.
A
It
could
be
a
conflicting
commit
right
if
we're
not
able
to
run
our
say
our
ci
pipelines
fast
enough,
so
that
it's
one
commit
per
pipeline,
which,
frankly
I
don't
see
a
lot
of
teams
with
with
significant
amounts
of
code
and
significant
amount
of
amounts
of
tests
that
are
able
to
you
know,
especially
as
you
get
kind
of
towards
the
end
of
a
of
a
sprint.
Let's
say
in
your
agile
process,
where
you
might
have
a
little
bit
of
a
storm
of
commits.
A
You
know
you
can
very
easily
have
one
commit
where
my
code
change
breaks
your
test
and
your
test
is
new.
My
code
is
new
and
you
know
there's
no
way
that
we're
going
to
discover
that
until
we
put
the
two
together
and
and
see
what
happens,
that
can
that's
easier
to
avoid
if
we
have
faster
cycles,
and
so
we
have
less
commits
per
cycle.
A
So
we
might
want
to
think
about
improving
our
cycle
time
or
maybe
you
know
staging
the
way
that
we
do
the
pipeline
so
that
we
run
you
know
tests
earlier
rather
than
later.
You
know
those
sorts
of
things
moving
on
to
service
level
objectives
around
fast
feedback,
the
fast
feedback,
fast
failure
outcome
mean
elapsed.
Time
for
for
a
successful
ci
cd
pipeline
is,
you
know,
kind
of
the
obvious
one
there
right
and
obviously
we
want
all
of
our
pipelines
to
be
to
be
instantaneous
and
all
code
should
have
zaro
boogs.
A
You
know
as
as
bugzilla
famously
referred
to
so
the
the
objectives
that
you
want
here
are
they're,
going
to
be
very,
very
contextual,
right,
they're,
going
to
be
based
on
the
size
of
the
application
or
system
the
amount
of
code,
we're
looking
at
the
amount
of
tests
that
you
run.
Is
this
continuous
integration?
A
Continuous
testing
is,
it
is
a
continuous
delivery.
So
it's
very
contextual
right.
You
know
what
you're
looking
for
here
be
be
aggressive
in
your
objectives
here
drive
for
zero.
You
know
drive
for
very
fast
drive
for
instantaneous.
You
know
for
some
teams,
anything
greater
than
15
minutes
is
going
to
be
obvious
or
sorry
is
it
going
to
be
awful
right
and
for
others,
if
you
get
below
15
hours,
they're
going
to
be
jumping
for
joy?
It's
very
contextual!
A
You
know.
Generally,
our
ci
pipelines
are
going
to
be
shorter
than
our
continuous
test.
Continuous
delivery
pipelines,
that's
kind
of
the
nature
of
the
beast
a
little
bit
and
then
you
know
mean
time
to
failure.
You
know
again,
there's
no
hard
and
fast
rule
here
or
no
hard
and
fast
objective
to
drive
for
so
maybe
start
with.
Let's
make
sure
that
most
of
our
failures
are
happening
in
the
first
half
of
the
pipeline
and
and
the
reason
for
this
is
really
around.
A
You
know
it's
really
around
testing
right.
So
you
know
if,
if
you
think
to
that,
that's
sort
of
the
image
of
of
the
test
pyramid,
you
know
in
terms
of
you-
want
to
have
most
of
your
tests
be
unit
tests
because
they
run
fast.
They
don't
need
a
lot
of
infrastructure
to
run.
In
fact,
they
really,
you
know,
really
need
no
infrastructure,
other
than
sort
of
the
cpu
and
the
code
to
run,
and
then
you
have
some
integration
tests
and
fewer
system
tests.
A
The
the
point
of
tracking
this
metric
is-
and
I've
observed
this
with
some
of
our
own
teams,
with
with
customer
teams
oftentimes.
What
an
engineer
will
do
is
they'll
only
write
a
system
test
for
a
new
piece
of
code
and
they
won't
necessarily
write
unit
tests.
Well,
that
can
be
bad
because
that
means,
if,
if
there's
a
failure,
if
there's
a
problem,
you
may
not
find
that
out
until
you're
90
of
the
way
through
the
pipeline.
A
Now,
if
that's
a
15-minute
ci
pipeline,
that's
not
so
bad,
but
it's
a
15-hour
long,
continuous
testing
with
lots
of
you
know:
security,
scans,
performance
testing,
scalability,
regression,
testing
acceptance,
testing,
staging
environments
involved,
deployments
involved.
You
know,
building
installers,
building,
docker
images
doing
deployments.
You
know,
helm,
charts
all
of
that
sort
of
business.
Having
that
failure
happen,
late
is
going
to
be
painful
so
and
one
of
the
things
I
always
encourage
our
engineering
teams
to
do
is
when
you
have
a
failure
in
a
pipeline,
and
it
happens
late
in
the
pipeline.
A
Ask
yourself
you
know:
could
I
have
written
a
unit
test
to
catch
this
problem
earlier?
You
know
what
was
there
a
way
you
know?
How
can
I
shift
left
that
failure,
because
the
sooner
it
fails,
the
sooner
we
can
fix
it
right
so
so
really
think
there
in
terms
of
you
know
kind
of
shifting.
You
know
the
objective
of
of
the
mean
time
to
failure
is
to
have
it
fail
as
far
left
in
the
pipeline
as
as
possible.
A
So
what
are
the
kinds
of
agreements
or
actions
that
we
would
that
we
put
in
place
here
around
our
stable
quality
signal
objective
so
on
the
success
to
failure
ratio
right
here?
Really
what
you're
looking
at
is
the
trend
right
at
the
at
the
at
the
extreme.
If
the
trend
is
up
and
to
the
right-
and
it's
not
looking
good
in
terms
of
the
failure
ratio,
the
the
percentage
of
of
your
pipelines
that
are
failing,
I
mean
you
may
need
to
stand
down
right.
A
You
may
need
to
go
to
that
part
that
painful
part
where
you
kind
of
pull
the
fire
alarm
and
say:
okay,
no
commits
that
aren't
related
to
fixing
the
current
set
of
problems
that
we're
experiencing
are
going
to
be
allowed.
We're.
You
know
we're
gonna,
it's
draconian,
sometimes
it's
necessary
when
you
have
a
failure.
It's
important
to
do
the
root
cause
analysis
on
that.
The
root
cause
analysis
here
is
not
about
pointing
fingers,
it's
very
important
to
to
to
do
that.
A
What
we're
trying
to
do
here
is
we're
just
looking
for
trends
right,
we're
looking
for
things
like
flaky
tests,
flaky
environments,
merge
problems.
You
know,
maybe
we
have
lots
of
commits
piling
up,
and
so
we
have
commit
conflicts,
and
maybe
that's
you
know,
driving
driving
the
the
trend
here.
If
you're
seeing
lots
of
flaky
tests
there's
a
few
things
you
can
do
here
right
most
of
our
most
of
the
test
tools
that
are
out
there
now
will
allow
you
to
rerun
a
test
if
it
fails
and
rerun
it
several
times.
A
You
need
to
treat
that
as
a
serious
problem
and
root
it
out
and
if
you
just
have
general
flakiness
in
in
the
infrastructure
or
in
tests,
then
you
might
need
to
go
to
that
sort
of
approach.
Where
run
10
pipelines
in
a
row
with
no
changes
collect
the
results
from
that
make
a
series
of
improvements
run.
A
So
if
I've
committed
something
into
a
pipeline,
I
need
to
be
watching
that
I
need
to
be
paying
attention
to
that
and
again,
you
know
the
size
of
the
team,
the
geographic
distribution
of
the
team,
whether
we're
following
the
sun
with
our
with
our
development
practices.
Those
sorts
of
things
are
all
play
into
this
right.
The
the
committers
here
should
should
root
cause
the
failures.
A
If,
if
you're
dealing
with
a
longer
pipeline,
are
you
able
to
restart
your
pipelines
at
the
point
of
failure?
You
know
if
you
don't
have
to
do
a
code
change
to
fix
a
pipeline
failure
if
you're
able
to
you
know
if
the
tooling
that
you're
that
you're
using
allows
you
to
restart
a
pipeline
at
an
earlier
stage
and
not
go
all
the
way
back
to
say
ci,
god
forbid,
to
in
order
to
run
your
pipeline.
A
Consider
that
and
if
you
can't
do
that,
maybe
what
you
want
to
do
is
the
action
you
want
to
take
is
to
invest
in
that
also
can
can
you
run
multiple
pipelines
in
parallel,
so
while
one
pipeline
is
running,
can
I
start
another
one
that
has
the
fix
and
can
I
use
elastic
infrastructure
in
order
to
take
to
take
advantage
of
that
and
then
again
on
the
root
cause
of
pipeline
failures?
I
you
know.
I
really
just
want
to
repeat
this.
If
you
have
flaky
tests,
if
you
have
flaky
infrastructure,
they
are
the
devil.
A
You
need
to
root
these
things
out
right
and
if
you
can't
fix
a
flaky
test
quickly,
disable
it
put
an
item
in
the
backlog
to
address
it
later
on,
but
do
not
let
flaky
tests
be
kind
of
the
russian
roulette
of
pipeline
success
because
that
will
end
in
tragedy
right
same
with
with
flaky
infrastructure.
If
you
can't
fix
your
flaky
infrastructure,
you
have
to,
you
might
have
to
commit
resources
or
people
to
it.
A
You
may
have
to
do
some
re-architecture
or
something
like
that,
and
then,
when
it
comes
to
things
like
bad,
commits
bad
merges
deployment
failures
and
so
on.
You
know
you
address
those
on
a
on
a
case-by-case
basis.
You
know
if,
if
you,
if
you're
tracking
a
lot
of
failures
coming
to
an
individual
or
a
team,
you
know
maybe
that
individual
needs
some
coaching.
Maybe
that
team
is
working
in
a
tricky
or
legacy
part
of
the
code
that
isn't
well
tested.
A
Maybe
we
need
more
testing,
more
unit
testing,
those
sorts
of
things
moving
on
to
to
our
service
level
agreements
or
actions
for
the
fast
feedback,
fast
failure
objectives
mean
elapsed
time.
This
is
the
frog
in
the
frying
pan,
you
know,
keep
a
keep
a
close
eye
on
that
right
because
you,
you
know
what
what
takes
five
minutes
today
takes
six
minutes
next
week,
seven
minutes
the
week
after
that,
and
then
before
you
know
it
you've
got
an
hour-long,
build
and-
and
you
don't
want
that
right-
you
don't
want
that.
A
There's
lots
of
tooling
available
out
there
to
compile
and
test
in
parallel.
You
know,
maven
gradle,
make
you
know
lots
of
products
out
there,
including
ones
from
cloudbees.
Of
course,
that
will
allow
you
to
run
your
pipelines
in
parallel
run.
Your
testing
and
pipeline
in
in
parallel
run
pipelines
in
parallel,
even
use
elastic
infrastructure,
so
that
you
can
do
this
all
of
those
kinds
of
things.
Look
for
the
longest
serial
chain
in
your
pipeline.
A
You
know
one
might
be
doing
something
that
we're
measuring
or
or
trying
to
test
with
the
other,
and
so
we
tend
to
very
often
run
system
tests
in
parallel.
Well,
if
you
could
throw
hardware
at
the
problem
and
fire
up
and
install
or
deploy
multiple
containers
with
your
system
in
it
with
your
application
or
your
service
in
it
and
then
run
all
of
those
tests
or
test
suites
in
parallel,
then
you
can.
You
can
dramatically
collapse
your
time.
A
We
we
just
did
this
exercise
in
one
of
our
product
teams
that
are
at
cloudbees,
and
we
were
able
to,
by
a
factor
of
between
four
to
eight
four
x,
four
to
eight
x
improvement
in
our
system
testing
time,
just
by
using
that
approach
of
of
deploying
the
application
to
multiple
instances
of
environments
in
parallel
and
then
running
those
tests.
In
parallel,
we
were
able
to
dramatically
collapse
the
length
of
our
of
our
system.
Test
pipelines
mean
to
mean
time
to
failure.
A
If
failures
are
happening
later
in
the
pipeline,
then
I
sort
of
talked
about
this
a
little
bit
earlier,
but
you
know
look
at
whether
these
are
system
test
and
integration
test
failures
and
really
what
you
ought
to
be
doing
is
maybe
writing
better
and
more
unit
tests.
Unit
tests
tend
to
run
earlier
in
pipelines.
A
They
tend
to
run
more
quickly,
you
you
can
write
more
of
them
and
really
you
know
again
what
I
encourage
our
teams
to
do
is
look
if
you,
if
you
end
up
with
a
system,
failure
system,
test
failure
or
even
an
integration
test
failure,
always
ask
yourself.
The
question
could
I
have
called
this
earlier
right.
Generally
speaking,
two
later
state
failures
are
going
to
be
more
tricky
to
address.
A
You
might
need
access
to
the
environment,
to
debug
or
diagnose
the
failure
right
again
if
integration
tests
or
system
tests
are
taking
too
long
to
run,
consider
breaking
them
up
into
smaller
batches
and
run
them
in
parallel
and
and
take
advantage
of
elastic
infrastructure
to
do
that,
take
advantage
of
being
able
to
deploy
your
applications
as
containers.
If
you
can
do
that
in
order
to
to
be
able
to
parallelize
and
collapse
the
cycle
times
here
and
find
the
problems
faster
again,
it's
not
just
about
how
long
is
is,
is
your
cycle
time,
but
really?
A
How
quickly
can
we
find
the
failures
and
because
that
impacts,
how
quickly
we
can
address
the
failure
and
how
quickly
we
can
kick
off
another
pipeline
that
will
address
the
the
problem
or
restart
an
existing
pipeline
at
that
at
the
failure
point.
So
thank
you.
Very
much
kind
of
in
summary
here
decide
on
your
desired
outcomes,
select
the
indicators
that
are
relevant
or
metrics
that
are
relevant
to
those
outcomes,
measure
against
those
objectives
and
then
have
agreements
or
actions
in
place
for
when
you
miss
those
objectives.
It's
really
that
sounds
very
simple.
A
I
know
it's
a
much
more
complex
thing
to
accomplish
than
that,
but,
but
really,
if
you
go
after
it,
that
way,
think
small,
think
incremental
improvements,
but
be
ambitious.
You
know
be
ambitious.
At
the
same
time,
you
will
be
able
to
control
and
drive
improvements
in
the
outcomes
in
in
in
your
pipelines
and
I'll
leave
you
here
with
a
with
a
screen
just
of
a
few
resources
here
that
were
useful
to
me
in
in
that
I
drew
from
in
putting
together
this
presentation.