►
From YouTube: DevOps Performance From a Different Dataset: What 30M Workflows Reveal - Michael Stahnke, CircleCI
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
DevOps Performance From a Different Dataset: What 30M Workflows Reveal - Michael Stahnke, CircleCI
What can we learn about DevOps best practices by looking at data from a SaaS with 900K dev users, 25K orgs and 30m+ builds a month, particularly when compared with surveys where users opt-in? Join CircleCI's VP of Platform, Mike Stahnke, to understand what this unique dataset uncovers about high performing engineering teams: - What trends stand out from the data? - Are they different than what is seen through surveys where responders opt-in to participating as compared to being aggregated through platform usage?
https://sched.co/ekB1
A
Hello,
everybody
I'm
michael
stonke.
I
run
the
platform
engineering
organization
at
circle
ci,
and
I
want
to
talk
about
what
we've
learned
from
looking
at
the
circle:
ci
data
about
how
teams
are
performing
so
I'm
mike
I'm
at
circleci.
I
guess
that's
the
intro,
that's
that's
the
simple
intro.
The
main
part
of
this
talk
we're
going
to
have
a
little
bit
of
a
setup
about
you
know.
What's
the
problem
set
we're
trying
to
ask
about?
A
What's
what
are
we
trying
to
learn
a
bit
about
the
data
diving
in
seeing
what
the
raw
data
tells
us
and
then
a
little
more
about
insight
about
how
we
look
at
that
data
and
what
we
can
derive
from
it?
What
we
can
maybe
hypothesize
about
it
and
and
talk
about
trends,
we're
seeing
in
the
industry
and
other
influencing
factors
so
beginning
with
the
setup.
There's
a
lot
of
devops
reports
out
there
that
talk
about
high
performing
teams
and
what
these
metrics
are
and
what
happens
and
to
me
those
have
always
been
fascinating.
A
They've
been
some
of
the
best
research
available
in
our
field.
I've
helped
author
a
number
of
these.
At
this
point,
I
worked
at
puppet
for
a
long
time
and
they
were
the
people
that
started
this
and
I
worked
with
them
on
a
number
of
these
questions
and
analysis
and
kind
of
how
teams
are
working
and
what
makes
a
team
high
performing
versus
a
team.
That's
maybe
medium
performing
or
a
team.
That's
less
successful,
and
you
know
you
ask
a
whole
bunch
of
questions
and
you
learn
a
whole
lot
about
the
way
teamwork
teams
work.
A
A
You
describe
your
performance
when
you're
filling
out
a
survey,
you
say:
well,
here's
how
we
behave.
Here's
how
my
team
behaves,
whereas
when
it's
derived,
I
can
say
when
I
look
at
the
data,
here's,
what
your
team
is
doing
and
here's
how
they're
actually
working
and
those
things
don't
always
agree
with
each
other
one
to
one
and
that
doesn't
mean
that
anybody's
lying
or
that
they're
not
telling
the
truth
or
that
they're.
You
know
perhaps
looking
at
something
with
rose-colored,
glasses
or
or
whatever.
A
It
means
that
sometimes
people
have
recency
bias
or
they
have
a
well.
I'm
not
thinking
about
this
other
project.
I'm
only
thinking
about
this
main
project
or
you
know
whatever
the
case
may
be,
and
so
we
kind
of
try
to.
I
guess,
normalize
out
that
data
a
little
bit
and
look
at
what's
really
going
on
org
wide
versus
just
maybe
a
specific
data
point.
A
So
in
this
data
set,
we
have
44
000
organizations
that
are
in
this
data
set
using
circle
ci
to
pull
from
160
000
projects,
and
that
makes
this
about
a
thousand
times
larger
than
any
state
of
devops
survey.
That's
existed
over
the
years,
and
so
this
is
just
a
much
much
larger
data
set,
which
should
mean
that
we
can
get
a
little
more
strong
signal
from
it
and
a
little
less.
A
You
know
if
there's
four
people
that
work
at
one
company
or
20
that
work
at
one
company
and
fill
out
something
that
they
can
skew
the
data
in
one
way
or
another.
I
don't
think
that
actually
happens
in
those
surveys,
because
again
the
data
analysis
is
very
thorough
and
professional
and
they
usually
can
sort
that
kind
of
thing
out.
But
this
is
just
a
larger
data
set
overall.
A
The
other
thing
that
we
can
do
this
year
is
look
at
what's
changed
year
over
year.
We
we
started
this.
This
question
set
last
year
where
we
were
asking
a
bunch
of
questions
in
our
data
science.
People
were
slicing
and
dicing
it
and
trying
to
figure
out
the
answers,
and
so
we
have
some
data
from
last
year,
and
even
this
year
we
re-ran
the
data
from
last
year
to
make
sure
that
we
had
processed
it
correctly
and
everything,
and
so
we
have
some
trends
that
we
can
see
of.
A
So
we
you
know,
we
do
30
days
increments
and
data
sets
and
for
some
of
the
trending
we'll
do
multiple
30-day
data
sets.
So
you
can
see
month
over
month
in
a
couple
spots.
You
know
our
organizations
we've
grown
a
little
bit
in
2020.
Our
number
of
projects
in
the
sample
grew
a
little
bit
in
2020
and
that's
basically
just
because
of
platform
growth
overall
for
circle.
Ci.
A
Traditionally,
there
are
four
metrics
that
are
used
in
devops
surveys,
as
kind
of
the
the
indicators
of
high
performing
teams
and
that's
your
deployment
frequency
your
recovery
from
failures,
your
change,
failure
rate
and
your
lead
time
to
change,
and
those
are
generally,
you
know,
kind
of
the
big
four
metrics
if
you're
high
in
those.
If
you
cluster
in
those
you
can
be
in
the
high
performing
group-
and
that
indicates
those
are
characteristics
of
high
performers,
okay,
and
so
how
do
we
look
at
that
now
if
you
map
that
into
a
ci
world?
A
Well,
if
we
have
deployment
frequency,
the
question
is
really:
how
often
do
you
initiate
your
work,
and
so
this
is,
how
often
do
you
initiate
a
pipeline
and
that
becomes
throughput
in
our
in
our
nomenclature?
Then
we
have
lead
time
to
change
that
has
what's
your
pipeline
duration
and
we
just
shorten
that
to
duration.
Your
change
failure
rate,
which
is
you
know,
your
pipeline
failure
rate
and
we
actually
go
to
success
rate
instead
of
failure
rate,
because
it's
more
fun
to
talk
about
success
than
failures
and
then
meantime
to
recovery.
A
A
So
throughput
is
what
we
start
with,
which
is
how
often
do
you
push
code
that
triggers
ci?
Most
projects
are
configured
to
run
and
push
to
upon
push
to
a
git
server.
Now
this
can
happen.
This
could
be
one
commit
push
to
a
git
server,
or
this
could
be
dozens
of
commits,
say
in
a
single
pull
request.
It's
you
know
a
lot
of
people
like
to
say
they
run
tests
on
commit,
but
usually
it's
actually
they
run
on
push,
and
so
with
throughput.
A
You
can
see
that
you
know
people
at
the
fifth
percentile
are
running
.03
jobs
per
day.
Well,
that's
not
very
many,
and
then
people
at
the
90th
percentile
are
kicking
off
a
workflow
16
times
per
day.
Okay,
that
makes
a
little
more
sense,
but
when
you
think
about
kind
of
the
narrative
that
you
know,
high
performing
teams
are
deploying
dozens
of
times
a
day
or
hundreds
of
times
a
day.
A
If
you
look
at
that
mean
at
the
bottom
there
and
that's
that's
just
running
running
a
throughput
that
might
not
even
be
a
deploy
every
time
that
could
be
just
doing
validation
or
code
pushes
so
people
are
not
pushing
code,
perhaps
as
often
as
they
talk
about
all
the
time
and
so
yeah.
This
analysis
is
kind
of.
Most
projects
are
not
doing
not
deploying
dozens
of
times
per
day.
So
why
does
that
become
such
a
popular
narrative?
Well,
a
lot
of
times
in
the
surveys.
A
They
ask
you
know:
what's
the
primary
application
or
service
you
work
on,
and
it
could
be
that
a
single
project
you
have
is
deployed
a
lot
or
it
could
be
that
your
single
application,
your
platform,
is
made
up
of
dozens
and
dozens
of
projects,
and
so,
while
any
one
of
those
projects
may
only
be
running,
you
know
one
to
eight
times
a
day
in
aggregate
you're,
still
deploying
60
or
70
times
a
day
and
for
instance,
if
you
were
going
to
look
at
circle
ci.
That
is
exactly
what
you
would
see.
A
Now,
if
you
look
at
duration,
that's
how
long
does
it
take
to
get
results,
and
so
this
is
basically
timed
after
code
push.
How
long
does
it
take
me
to
get
signal
on?
What's
going
on
for
my
code,
five
percent
of
our
builds
finish
in
less
than
12
seconds
and
when
I
say
five
percent,
that's
that's
about
500
000
in
the
sample
of
of
this
exact
sample
that
I
pulled,
and
so
that's
not
a
small
amount,
but
less
than
12
seconds.
You
think
what
happens
in
12
seconds.
A
Well,
it
could
be
a
number
of
things.
It
could
be.
You
get
a
very
quick
failure.
It
could
be.
The
entire
workflow
is
something
like
I'm
putting
a
a
static
file
onto
say,
an
s3
bucket
or
I'm
you
know,
modifying
some
documentation.
It
could
be
very
simple
things.
It
could
be
your
running
unit
tests
that
have
been
really
well
optimized,
but
five
percent
of
our
builds
are
finishing
in
less
than
12
seconds,
but
then
you
can
see
that
a
lot
of
our
builds.
A
You
know
they
increase
quite
a
bit
from
there,
but
50
of
the
bills
are
under
that
four
minute
mark
and
then
the
95th
percentile
you're
at
34
minutes
and
our
average
is
right
around
25
minutes,
because
you
know
you
start
looking
at
the
outliers
that
are
in
there
and
pulling
that
number
up.
I
think
most
of
most
of
our
workflows
time
out
after
about
five
hours,
so
that's
about
the
longest.
A
And
so,
if
things
take
longer
than
four
or
five
minutes,
they
may
throw
a
flag
and
say
hey
it's
time
to
re-engineer
our
test
suite,
because
what
matters
to
us
is
cycle
time
and
how
fast
we
can
give
signal
back
to
developers,
and
so
some
teams
work
very
very
hard
to
keep
that
build
time
under
you
know
four
or
five
minutes
and
other
teams
say:
hey
25
or
30.
Minutes
is
great.
For
me,
I
know
at
a
place
I
used
to
work.
A
25
or
30
minutes
would
have
been
out
would
have
been
outstanding
to
get
signal
back,
because
sometimes
it
was
seven
or
eight
hours,
and
so
you
know
it
all
depends
on
your
perspective
about
what
you're
doing
how
detailed
and
thorough
your
tests
are.
You
know
maybe
the
number
of
platforms
or
permutations
you're
testing
things
like
that.
A
So
in
2020
you
can
see
that
there's
most
the
average
durations
or
the
durations
that
across
the
percentiles
are
a
little
bit
larger
in
2020,
and
that
might
be
that
there's
more
tests
happening,
perhaps
more
investment
into
validation
in
certain
segments.
Maybe
more
complicated,
workflows
things
like
that,
but
there
are
a
little
more
usage
going
on
in
2020
than
there
was
in
2019..
A
A
Next
is
success
rate,
so
how
often
does
your
pipeline
complete,
complete
with
a
green
status,
is
more
or
less
what
this
this
metric
is
trying
to
measure
and,
as
you
can
see,
we
have
in
the
in
the
lower
end
of
the
percentiles
zero
it
never
returns,
green
and
in
the
higher
ends
100,
where
it's
always
returning
green
and
so
some
of
our
data
samples
dabble
with
ci,
but
they
don't
necessarily
get
a
working
build.
This
could
be
somebody
starting
with
their
own
individual
project.
It
could
be
a
side
tool,
it
could
be.
A
A
They
have
few
tests
that
maybe
they
have
tests
that
just
say
return
true,
so
they
can
never
fail
and
we
see
all
those
types
of
things
in
our
workflows,
but
you
do
see
that
some
tests
or
some
organizations
have
workflows
that
never
fail,
and
so
what
you
look
at
overall
is
the
the
overall
success
rate
on
average.
Is
you
know
it's
54
in
the
mean,
and
at
the
50th
percentile
is
61,
and
what
I
found
really
interesting.
Was
this
really
didn't
change
from
2019
at
all?
A
You
can
see
there's
a
one
percent
difference
at
the
50th
percentile,
but
we
wanted
to
look
into
this
a
little
bit
more
and
see.
Well,
okay,
is
that
true?
Is
this
really
that
static
year
over
year?
And
if
you
do
look
at
the
little
bit,
the
50th,
75th
and
85th
percentile
that
things
are
increasing
a
little
bit,
so
the
people
that
are
using,
I
guess
at
that
percentile-
are
have
increased
their
behavior
a
little
bit
they're
having
a
little
bit
higher
success
rates
than
what
they
were
a
year
ago.
A
A
In
terms
of
you
know,
the
2020
impact
here
recovery
time
went
down
in
every
category
and
one
of
the
things
that
I
wonder
about
is
is
that
because
more
people
are
at
home
and
more
people
are
paying
attention.
Is
that?
Because
there
are
fewer
distractions,
I'm
not
positive
on
that,
but
it
does.
Look
like
the
2020
is
slightly
better
than
2019
was
in
terms
of
how
long
pipelines
sit
in
their
red
state.
A
Yeah,
so
they
have
improved
year
over
year,
and
so
now
I
want
to
really
get
to
the
insights,
which
I
think
is
the
most
interesting
part
of
this.
Rather
than
just
read
your
data
on
a
screen.
Let's
talk
about
what
this
data
tells
us
a
little
bit
more,
so
what
development
practices
definitely
work
like?
What
can
we
pull
out
of
this?
Well,
I
can
tell
you
that
success
rate
does
not
correlate
with
company
size
at
all.
We
looked
at
company
sizes
and
and
team
sizes
and
org
sizes,
and
tried
to
look
at
okay.
A
A
A
I
guess
that
that's
always
the
longest
that
we
see
recovery
time
decreases
with
an
increased
team
size
and
so
the
more
people
looking
at
the
build
the
faster
your
recovery
time
is,
which
makes
sense.
But
it's
only
up
to
a
point
and
up
to
that
point
was
around
200,
which
was
actually
much
larger
than
I
would.
I
would
have
guessed,
but
the
vast
majority
of
our
data
samples
sits
there
with
teams
between
kind
of
that
five
to
nine
mark
and
not
much
above
that.
A
Every
now
and
then
you
get
projects
that
are
maybe
the
the
core
project
that
many
teams
contribute
to,
and
so
you
get
many
more
contributors.
You
know
where
it
could
be
in
the
hundreds,
because
perhaps
the
entire
engineering
department,
or
even
people
outside
the
engineering
department
still
contribute
to
the
main
core
project,
but
that
sample
is
a
little
bit
smaller
than
kind
of
just
the
across
the
board.
A
All
the
different
projects,
the
longest
recovery
times,
are
again
from
teams
of
one
and
so
again,
if
you're
working
on
this
yourself,
maybe
you
maybe
you're
working
on
your
own
tool.
You
get
a
red
build
and
you
think
well,
whatever
I'll
work
on
it
later,
and
that
could
be
ten
days
later.
It
could
be
two
weeks
later.
It
could
be.
You
know
three
hours
later.
It
just
depends
on
kind
of
when
that
fits
back
into
your
schedule.
A
A
If
you
have
a
team,
your
team
is
probably
going
to
perform
better,
and
that
does
make
sense,
but
if
you're
chewing
things
on
your
own,
basically
it
doesn't
look
like
you
will
have
as
good
of
outcomes,
and
so
I
guess
one
of
the
things
that
tells
me
is
you
shouldn't
be
that
10x
developer,
who
sits
in
a
corner
and
says
I
can
do
this
all
myself,
because
you
won't
actually
have
as
good
of
outcomes,
and
we
can.
We
can
show
that
based
on
the
data
sets.
A
The
next
question
that
I
asked
was
is:
is
this
don't
deploy
on
friday?
A
real
thing:
this
has
been
kind
of
an
old
school
level
of
thinking.
I
think
that
goes
back.
You
know
to
the
classic
operations
days
before
devops
was
a
big
movement
was
well,
we
don't
make
changes
at
certain
times.
That's
the
best
way
to
be
stable
and
don't
deploy
on
a
friday
became
quite
a
real
thing
overall,
and
so
I
wanted
to
see.
Could
I
see
that
in
any
of
our
data?
A
Well,
we
we
see
70
less
throughput
on
weekends,
and
that
means
that
there's
a
whole
lot
less
happening
on
our
on
circle
ci
on
a
weekend
than
there
is
during
the
week,
and
it
also
means
my
aws
bill
goes
down
quite
a
bit.
So
that's
that's
something
I
usually
look
forward
to,
but
then
on
on
fridays
we
see
11
less
throughput
than
we
would
say
comparative
to
tuesday,
wednesday
or
thursday,
and
I
pull
off
tuesday,
wednesday,
thursday,
intentionally
because
of
the
middle
of
the
week
and
internationally.
A
You
know
a
lot
of
times.
People
do
take
fridays
and
mondays
off.
If
they're,
trying
to
you
know,
extend
a
long
weekend
and
that
probably
accounts
for
most
of
that
drop.
The
other
thing
that
does
account
for
that
drop
is
that
mondays
and
fridays
are,
you
know
again,
different
time
zones
around
the
world.
You
know
we
use
utc
for
all
of
our
logs
and
all
of
our
data
gathering
and
stuff
like
that.
But
you
know
at
midnight
utc
monday
morning
is
still
sunday
evening
in
north
america.
A
A
Another
interesting
set
of
questions
that
I
love
to
ask
is,
is
what
do
we
see
about
languages
and
language
trends
going
on
in
this
data
set,
you
know,
are:
are
there
certain
languages
that
are
that
lend
themselves
to
better
outcomes
across
the
ci
system
across
teams?
And,
generally
I
say,
take
these
with
a
grain
of
salt
because
anytime
I
put
up
a
chart
of
languages.
A
It's
it's
half
of
it's
just
to
provoke
you
to
think
about
what
your,
what
your
language
choices
are,
and
why
and
the
other
half
is
to
make
sure
that
you
can
argue
with
your
other
developed
and
friends
about
what
language
choices
you
made
and
why
so
languages
in
our
sample.
When
you
look
at
you
know
the
giant
sample
that
I
said
that
was
160
000
projects,
here's
basically
the
breakdown
anything
below
0.83
just
wasn't
even
counted,
because
this
is
the
top
20..
A
But
you
know
we
have
a
lot
of
dynamic
languages.
We
have
a
lot
of
statically
typed
languages.
We
have
a
lot
of
more
markup
and
markdown
type
languages.
So
there's
a
you
know
several
different
styles
of
language
in
our
sample
here
on
throughput.
This
is
basically
how
often
you're
running
on
circle.
Ci,
you
know,
ruby
is
our
is
the
thing
that's
happening,
the
most
often
that
could
be
because
their
durations
are
rather
slow
or
rather
quickly.
You
get
feedback
very
quickly.
A
It
could
be
there's
just
a
lot
of
ruby
projects,
it
could
be
there's
a
number
of
different
things,
and
you
can
see
that
this
is
just
how
often
different
things
happen.
Now
the
dockerfile
being
at
20
was
a
little
bit
surprising
to
me,
because
we
do
have
a
lot
of
people
that
build
docker
images
via
ci,
but
it
could
also
be
that
that
doesn't
just
get
kicked
off
that
often
it
happens
less
often
than
than
a
lot
of
the
other
software
projects.
A
I
guess
that's
because
dockers
would
be
the
final
artifact
versus
perhaps
an
intermediate
artifact
of
you
know,
compiling
your
go
program
and
then
placing
a
docker
docker
container
things
like
that
on
the
success
rate
at
the
50th
percentile.
This
is
the
order.
I
I
find
it
interesting
that
vue,
css
and
shells
show
up
at
the
top.
I
don't
see
a
lot
of
tests
for
css
and
even
in
shell,
you
don't
see
a
lot
of
tests,
usually
it's
kind
of
just
actions,
and
so
those
actions
may
succeed,
regardless
of
kind
of
what
happens.
A
A
lot
of
shell
exits
zero.
Unless
you're
very
intentional
about
how
you
write
things.
Some
people
probably
do
use.
You
know
like
a
bash,
automated
test,
suite
bats
or
whatever,
but
most
people,
I
think,
are
just
using
shell
to
complete
a
workflow,
maybe
copy
a
file
over
there,
and
you
know
it's
going
to
return
zero
regardless
in
a
lot
of
cases,
and
then
dockerfile
also
usually
succeeds
in
that
you
build
a
docker
container.
A
Whether
or
not
the
container
was
what
you
wanted
or
not,
maybe
something
that
you
need
to
requires
further
introspection,
but
the
first
real
language
that
I
think
that
has
a
lot
of
testing
up
and
down
all
the
way
through
would
be
probably
go
at
number
eight,
and
what
I
find
fascinating
here
is
that's
the
first.
You
know
that's
a
strongly
typed
language
and
then
you
get
into
many
dynamically
typed
languages,
and
then
you
get
back
into
the
rest
of
statically
typed
languages
lower,
and
so
you
see
you
know.
A
A
The
longer
things
the
things
are
more
at
the
bottom,
the
recovery
time
being
slower
for
some
some
of
the
stacky
type
stuff
it
that
makes
sense
to
me
because
their
durations
are
a
little
bit
longer
and
a
lot
of
times.
Your
recovery
time
is
longer
if
your
duration's
longer,
if
you
know
your
pipeline,
is
going
to
take
35
minutes
to
run.
A
You
may
step
away
from
your
desk.
You
may
go
to
another
call
another
meeting,
and
then
you
don't
see
the
result
for
a
while
or
you
don't
get
back
to
working
on
the
result
for
a
while,
whereas
if
you're
going
to
get
your
results
in
three
minutes,
you
may
be
able
to
correct
them
very
quickly,
and
so
your
correction
time
will
also
kind
of
correlate
with
that
in
terms
of
duration.
Things
that
happen.
A
A
I
can
tell
you
that
php
showed
up
in
a
really
good
light
in
most
of
these
metrics,
which
was
really
odd
to
me,
and
I
remember
making
recommendations
of
everybody
using
php,
which
I
didn't
believe,
but
thought
was
rather
amusing
and
this
year
it
shows
up
much
more
like
how
I
expect
it
to
group
next
to
python
and
near
ruby
and
things
like
that.
So
this
year
the
data
seems
to
be
a
little
clearer
on
how
these
all
group
together.
A
One
of
the
other
things
we
talked
about
was
branch
information,
because
we,
you
know
a
lot
of
times,
people
build
off
a
auto
feature
branch
and
then
they
merge
in
the
main
line.
How
does
that
all
work-
and
the
first
question
that
came
up
to
me
for
2020-
was:
have
people
really
worked
on
renaming
their
branches
from
master?
A
So
did
this
actually
decrease
this
year?
The
answer
is
no.
It
really
didn't
in
any
significant
way-
and
I
put
yet
in
here-
because
I
don't
think
github
has
completed
all
the
changes
to
make
it
so
that
any
new
project
doesn't
have
master
or
maybe
they
did
with
new
projects,
but
they
haven't
done
anything
retrofitting
existing
projects.
A
A
Another
branch
branch
topic
is
that
teams
are
innovating
and
experimenting
on
feature
branches,
and
so
you
can
tell
this
because
the
success
rate
on
the
default
branch
is
much
higher
than
on
non-default
branches,
and
so
the
default
branch
in
a
lot
of
cases
is
mainline
or
main
or
master
and
the
the
success
rate
there
is
significantly
higher
than
what
it
would
be
on
some
of
those
branches
that
lead
into
it.
You
know
the
branches,
you
would
either
be
making
a
pull
request
or
a
merge
request
on
things
like
that.
A
So
we
have
about
an
eighty
percent
success
rate
on
on
the
on
the
main
branch
versus
a
and
it's
a
hundred
percent
above
the
75th
percentile.
So
basically,
success
is
happening.
A
whole
lot
on
the
mainline
branch,
whereas
it's
only
58
for
that
non-default
branch
and
what
that
tells
me
is
people
are
using
the
ci
system
to
get
signal
at
times
and
so
maybe
they're
merging
a
thing
where
they're
like
I'm,
not
sure
this
is
going
to
work,
I'm
not
positive,
but
I'll.
A
Let
the
ci
system
tell
me
if
this
works
or
it'll
run,
maybe
a
complicated
integration
test
that
I
just
can't
simulate
on
my
workstation
too
easily,
but
either
way.
What
this
really
supports
is
that
things
are
happening
outside
of
mainline
they're
getting
merged
into
mainline,
and
that's
how
we're
figuring
that's
how
people
are
getting
signal
and
they're,
also
taking
really
good
care
to
keep
mainline
clean
and
keeping
it
green,
which
is
really
great
when
you
have
multiple
developers
working
on
something.
This
is
just
following.
You
know
the
the
practices
of
trunk
based
development.
A
They
really
want
the
trunk
to
be
clean.
All
the
way
through
the
duration
on
the
default
branch
is
faster
at
every
single
percentile
than
it
is
on
on
the
feature
branches
by
default
or
by
on
feature
branches
at
large,
and
so
what
this
basically
means
is
is
branches
that
are
not
in
mainline
generally
finish,
slower
than
branches
that
are
mainline
and
again
that
makes
sense,
because
there's
new
experimentation
going
on,
perhaps
your
tests
are
flaky.
Perhaps
they
need
to
be
optimized,
but
as
they
get
into
mainline,
that
happens
and
also
recovery
time
is
lower.
A
A
A
What
has
that
really
told
us
from
our
data
set
and,
as
you
start
to
graph
thing,
we
pulled
graphs
from
2019
all
the
way
through
2020,
and
we
we
try
to
do
you
know
30
day
increments,
on
samples,
but,
as
you
can
see,
there
are
dips
that
are
noticeable
in
the
march
and
april
time
frame
or
changes.
I
guess
I
would
say
in
this
case
throughput
went
up
in
march
and
april.
A
We
see
that
you
know
a
lot
of
things
went
up
in
march
and
april
and
to
me
the
way
that
I
I
thought
about
this
and
tried
to
rationalize
this
was
you
know,
perhaps
there's
a
bunch
of
developers
that
were
at
conferences
that
were
on
vacation
that
were
away
from
the
computer.
I
guess
is
probably
the
right
way
to
put
it
that
returned
to
the
computer
in
march
and
they
started
working
on
things,
maybe
they're,
adding
more
testing.
A
A
So
you
see
duration
increased
a
little
bit
in
february
and
then
that
increase
accelerated
in
march
and
then
decreased
in
april,
which
I
think
you
know.
We
can
basically
say
that
the
hypothesis
here
is
that
more
tests
were
happening
in
march
driving
up
the
duration
and
in
april
there
was
more
of
a
concentrated
effort
on
optimization
and
that
kind
of
adds
up
when
you
think
about
the
way
that
people
were
kind
of
sheltering
in
their
homes.
They
were
working
on
things
and
then
it
was
like
wow
we're
adjusting
to
this.
A
Okay,
I'm
going
to
work
on
this.
I'm
going
to
make
this
a
real
thing,
I'm
going
to
make
it
good
success
rates
also
increased
in
you
know.
During
the
pandemic,
as
everybody
went,
went
home
and
sheltered
in
place,
they
got
higher
success
rates
and
part
of
that
again
could
be
that
they
were
paying
more
attention
to
the
direct
things
that
their
company
was
doing.
They
weren't
on
the
road
they
weren't
as
distracted,
they
weren't
as
sidetracked.
A
I
do
love
the
25th
percentile
just
falling
off
there,
and
I
wonder
if
that's
just
like
20
20
in
a
graph,
it's
just
like.
No,
I
can't
anymore,
or
if
it's
you
know,
those
are
projects
that
are
not
core
to
the
business
and
so
they're
just
being
left
alone,
and
maybe
they
failed
and
somebody
was
like
well,
I
just
don't
have
time
to
give
on
that
anymore,
and
so
I'm
not
going
to
you
know,
there's
hypotheses
there
and
I
we
haven't.
I
I
don't
have
a
a
clear
narrative
there.
A
My
hypothesis
is
that
that
is
non-core,
business
inside
projects
and
tools
that
were
just
not
essential
and
they're
no
longer
getting
the
attention
they
once
got.
The
success
rates
were
the
highest
on
record
for
april
of
2020..
Again,
that's
kind
of
when
you
had
full
workforce
at
home,
fully
engaged
getting
longer
days.
Things
like
that,
so
the
hypothesis
there
is
that
people
were
working
hard
on
core
business
stability
in
in
march
and
april,
a
lot
of
companies
were
going
through,
reject
reduction
in
workforce
and
other
type
of
economic.
A
You
know
things
where
they
were
like
wow
this
this
little
side
thing.
I
don't
really
want
to
dabble
with
that
anymore.
Let's
make
sure
the
core
of
our
business
is
clean
and
works
and
is
stable,
and
I
think
we
saw
that
show
itself
through
our
data
from
developers
and
then
recovery
time.
You
know
we
can
see
that
it
hasn't
changed
significantly
throughout
the
pandemic
period,
but
it
did
drop
a
little
bit
and
we
saw
that
in
some
of
the
2019
to
2020
data.
A
Earlier
you
know,
generally,
the
the
95th
percentile
has
dropped
quite
a
bit.
Some
of
the
people
that
were
really
good
are
still
really
good
at
their
recovery
times,
but
recovery
time
has
been
improving
since
april
and
orrs
with
the
longest
recovery
time
have
definitely
improved
and
that
that
was
kind
of
indicated
by
this
top
line
that
green
bar
there,
where,
when
you
had
the
longest
recovery
time
that
has
increased
or
decreased
significantly
and
again.
A
This
is
a
this
is
a
golf
not
bowling
thing
where
you
want
to
be
the
lowest
score
to
win,
so
people
are
recovering
faster
and
faster
than
what
they
used
to,
and
the
hypothesis
here
is
that
people
have
fewer
distractions.
When
they're
working
at
home,
they
don't
get
pulled
into
meetings
or
go
to
lunch
or
go
grab
a
coffee
and
then
forget
about
it
and
then
come
back.
Oh,
I
was
working
in
that
pipeline.
I
should
probably
go
fix.
A
It
there's
probably
a
little
bit
less
of
that
happening,
but
when
I
say
fewer
distractions,
it's
of
course
for
some
values
of
distraction.
Some
of
us
have
very
new
distractions
to
work
with
whether
that's
children
at
home
or
taking
care
of
loved
ones
or
worrying
about
sickness
or
just
doom-scrolling
all
day
about.
What's
going
on,
there's
there's
a
lot
that
can
be
distracting,
but
I
do
think
that
people
generally
are
a
little
bit
more
in
front
of
the
computer
and
paying
a
little
more
attention
to
what
their
their
signals
are
in
their
delivery.
A
A
The
more
people
you
have
working
on
this
stuff,
the
better
you'll
be
because
I
you
know
part
of
it
to
me
is-
I
think
you
just
care
about
your
your
developer
experience
for
other
developers
if
they
have
to
wait
10
minutes
for
tests
when
they
can
rate
five
people
work
on
it,
make
sure
they're
only
waiting
five
and
if
you
know
the
pipeline's
red
well,
somebody
else
needs
it
to
be
green
to
do
their
work.
You're
going
to
make
sure
it's
green,
and
I
think
that
adding
collaborators
is
a
really
great
thing.
A
So
I
hope
you
found
this
interesting.
You
know
it's
definitely
a
dive
into
the
data.
This
is
kind
of
a
cursory
survey
of
the
whole
set
of
data.
Because
there's
a
lot
you
can
go
into
there's
a
lot
of
questions.
You
can
dig
into
there's
a
lot
of
questions.
You
can
ask
the
data
team,
they
can
slice
and
dice.
A
But
if
this
is
interesting
to
you,
I
will
remind
everybody.
We
are
hiring.
We
have
been
hiring
like
mad
over
at
circle
ci
and
we
still
have
many
many
positions
open
in
engineering
and
data.
So
if
this
stuff
is
interesting
to
you,
you're
welcome
to
apply,
and
I
just
want
to
say
thank
you.
I'm
again,
I'm
michael
stonke,
I'm
the
vp
of
platform
over
at
circleci.
A
You
can
find
me
at
stonema
on
basically
any
internet
service
that
you
care
to
find
me
on,
and
I
also
want
to
give
a
special
thanks
to
ron
and
melissa
who
helped
gather
all
this
data
and
sliced
it
and
diced
it.
To
answer
a
lot
of
my
questions
about
it.
I
did
not
do
this
all
on
my
own.
I
am
not
a
data
scientist,
but
luckily
I
know
people
who
are
so.
Thank
you
so
much.