►
From YouTube: Verify:Testing Internal Customer call - 2021-05-27
Description
Today we talked about the impact of Code Quality moving to the Static Analysis group, what the Engineering Productivity group is working on next and the best way to reduce time between build failure and start of the next pipeline.
Agenda: https://docs.google.com/document/d/1ijjjvLeIY6a82dS2_8H-G0bhvpaZ5zxPgYpU08EYaHc/edit#heading=h.b4f8595pqvm6
A
This
is
the
internal
customer
call
for
the
testing
group
may
27
2021.,
I'm
going
to
kick
off
the
agenda
real,
quick,
there's,
a
couple
of
change,
a
couple
of
changes
in
the
roadmap
deck
things
of
note.
Code
quality
is
being
handed
off
as
a
category
over
to
the
static
analysis
team,
the
customer,
the
vision,
the
roadmap,
the
feature
set
all
aligns
really
really
nicely
with
what
they're
going
to
go,
build
next
and
some
of
what
they've
already
built.
A
A
So
it
can
be
read
in
all
of
our
features,
kind
of
as
a
proof
of
concept
for
customers
who
want
to
run
without
docker
and
docker,
so
they
could
use
things
like
super
winter
or
other
linters
instead
of
the
single
code.
Climate
part
of
that
is
another
issue
that
we'll
deliver
in
14
1
that
allows
multiple
reports
across
jobs
to
be
pulled
together
as
long
as
they're
the
code,
climate.json
format
and
all
be
pulled
into
our
feature
set.
A
So
the
mrdiff
sorry,
the
mr
widget
already
utilizes
that
today,
but
the
full
report
in
the
diff
will
utilize
that
after
14
1,
so
once
we've
delivered
that
we'll
finish
up
that
handoff
and
refocus
our
efforts
on
performance
testing.
Hopefully
so
that
was
in
there
talked
about
code
climate.
Handing
off.
I
was
really
excited
about
code
quality
being
handed
off.
Apparently
I
made
it
to
agenda
items
wanted
to
make
sure
everybody
knew
and
then
dog
fooding
items
that
does
or
will
wrap
up
the
code
quality
in
the
mrdiffs.
A
A
B
I
will
track
it
down
because
I
think
I've
put
it
in
about
four
different
one-on-one
docs
throughout
the
week
yeah.
So
it
came
up
in
the
key
review
and
if
you,
if
you
watch
that
I'll
say
very
great
discussion
between
sid
and
myself,
where,
where
it's,
it's
pretty
clear
that
we're
measuring
the
something
that
is
not
what
sid
thinks
is
the
right
measurement
and
and
he's
he's
a
little
disappointed
that
we
haven't.
B
B
I
think
this
is
something
I'd
like
to
at
least
have
the
team
make
some
progress
on
over
the
next
few
milestones,
probably
14-1
when
I
say
the
team
ep
team
to
be
clear
here,
because
I
don't
think
when
I
looked
at
the
road
map
deck
it
didn't
seem
like
there
was
any
interest
from
your
team
in
building
a
feature
around
this
james.
I
guess
it
seemed
to
me
that
us
proving
out
some
solution
like
us,
proving
the
value
was
more
where
we
had
left
this
feature
off
is.
Was
that
a
fair
summary.
A
Yeah,
I
think
we
haven't
seen
customer
demand
for
this
as
much
as
we
have
for
identification
of
flaky
tests
like
when
it
comes
to
analyzing.
The
previous
test
run
where
we're
seeing
the
customer
demand
is
around
show
me
that
this
is
a
flaky
test,
so
I
can
go
fix
it.
It
doesn't
cause
a
lot
of
strife
in
the
future
versus
I
just
went
and
fixed
this
test
run
at
first
to
make
sure
I
got
my
fix
right
and
then
around
the
rest
of
the
pipeline,
which
it
seems
like
that's
how
I
parse
apart.
B
So
there
were
two
interesting
things
in
the
discussion
that
were
new
nuggets
for
me,
so
some
of
the
things
that
I
think
I'll
say
where
new
information
is
we're.
Not
we
used
to
talk
about
this
with
respect
to
short
circuiting
the
pipeline
and
stopping
everything
we
we're
not
doing
that.
It's
just
run
it
and
see
what
information
we
can
do
and
see
how
maybe
other
productivity
indicators
may
adjust
with
respect
to
this
information
being
surfaced
to
engineers.
B
A
B
We
don't
want
an
indicator
to
train
people
to
like
look
at
your
pipeline.
That's
running,
don't
do
anything
else
and
then
act
when
you
see
a
failure.
So
we
geared
on
pipeline
duration,
we're
going
to
shift
that
that
indicator
and
with
the
experiment
we
want
to
just
wow,
never
mind.
I
just
lost
sight
of
your
question
james,
I'm
sorry.
It's
been
kind
of
a
hectic
day.
What
was
your
question
holy
well.
A
So,
and
maybe
I
missed
something
I
what
I
heard
previously
was
that
we
will
not
short-circuit
the
pipeline,
so
so.
The
way
that
I
understand
this
is
that
we're
going
to
create
a
new
job
that
runs
all
of
the
tests
that
failed
previously
run
those
first.
If
all
of
those
pass
great,
the
rest
of
your
pipeline
is
passing,
but
even
if
they,
if
they
fail,
then
the
pipeline
should
fail,
but
will
it
also
short-circuit
the
rest
of
the
jobs
in
flight
and
say
hey?
B
It
will
not
it's
it's
essentially
just
gonna.
It's
gonna
try
to
be
an
early
signal
that
it
they've
they
failed.
Ideally,
that's
what
I'm
saying
josh
joshua
was
on
co
shadow
and
he
raised
the
question
when
I,
when
I
was
talking
about
we
we're
not
going
to
short
circuit
the
pipeline.
We
geared
the
kpi
on
pipeline
duration
because
developers
aren't
notified.
He
said
he
asked.
Oh
there's
no
build
notification,
I
don't
know
if
we
have
a
feature
for
that
I'll
create
an
issue.
B
That's
where
there
may
be
product
notification
like
there
may
be
a
change
in
the
product
features.
I
don't
know
ultimately
where
this
experiment
is
going
to
go.
To
be
honest,
I
just
think
it's
something
that
we're
going
to
try
to
do
and
look
at
other
productivity
indicators
to
see
what
we're
going
to
glean.
A
B
I
want
to
see
the
effect
on
mean
time
to
merge
and
the
number
of
pipelines
that
are
run
to
merge.
Mrs
I'll
say
I
am
not
convinced
there
is
an
effect,
but
there
are
other
stakeholders
who
are
looking
at
this
metric
who
think
there
is
going
to
be
an
effect
on
those
metrics,
and
I
think
it
is
because
of
those
stakeholders
believe
there's
an
effect.
I
believe
that
it's
worth
doing
an
experiment
to
see.
I'm
not
I
I'm
not.
C
Sorry,
I
was
just
I
and
tell
me,
I
may
have
missed
something.
I've
been
like
I've
been
trying
to
follow
along
with
like
what
were
have
been
versus
want
to
look
at.
I
just
thought
of
is:
would
it
be
an
interesting
metric
to
look
at
the
time
between
a
build
failure
and
then
and
a
new
pipeline
created
on
that
branch?.
B
B
C
B
Other
the
I'll
say
like
the
the
metric
that
I
was
going
to
look
at
to
try
to
measure
something
similar
was
like
number
of
canceled
pipelines
per
mr,
because,
like
that's
where
that's
where
like,
if
someone's
looking
at
that,
mr
widget,
they
may
just
push
something
up
and
just
be
like.
Oh,
I
see
that
already
failed,
I'm
just
going
to
commit
push
and
then
it.
D
B
C
A
D
A
D
A
B
B
This
is
where
I'm
kind
of
like.
To
be
honest.
This
is
where
this
is
where
I'm
saying
like
I'm
kind
of
torn,
because
I
feel
that
it
takes
developers
out
of
the
flow
state
where,
if
they
context
switch
into
something
else,
I
almost
feel
that
some
of
the
like
this
is
where
our
metrics
could
kind
of
may
work
for
some
developers
and
not
for
others,
like
some
people
might
just
want
to
like
tunnel
vision
on
one
thing
and
see
it
all.
The
way
through
other
people
may
want
to
push
five
things
forward.
A
Two
types
of
workflows:
one
is
I'm
just
going
to
stare
at
the
pipeline
anyway,
because
I
want
to
iterate
through
this
as
fast
as
I
can,
or
I
might
even
create
a
custom
job
that
just
runs
these
three
tests
and
that
is
already
part
of
my
workflow
so
that
I
can
iterate.
I
am
on
the
seminar
until
it
is
done
or
the
person
who
just
they
run
the
standard
pipeline
and
completely
contact
switching
to
something
else,
because
they
know
either
it's
going
to
be
green
after
an
hour
or
it's
going
to
fail.
A
But
now
they
have
an
hour
of
time
or
even
overnight
that
they
can
then
move
on
to
something
else.
And
this
creates
almost
that
different
you're
going
to
get
a
failure,
but
maybe
in
like
20
minutes
and
if
you
were
already
staring
at
the
pipeline
you're
going
to
know
just
as
soon
as
you
would
have
before.
If
you
weren't
then
you're
going
to
get
a
notification
that
potentially
pulls
you
out
of
the
flow
of
the
other
thing.
D
Yeah,
hey
kyle,
to
your
comment
too.
I
I
think
the
the
two
different
ways
to
look
at
it,
that
being
narrow
focus
versus
multitasking
in
effect,
is
that
it
depends
really
on
cognitive,
low.
You
know
a
great
degree
because
some
of
our
developers
they
they
can
do
that
because
they
know
all
the
things
they're
working
on
like
the
back
of
their
hands,
they've
been
in
it
for
years,
and
it's
easy
for
them
to
switch
from
one
to
the
other.
So
I
you
are
going
to
have
a
variance
between
developers.
A
And
I
think
if,
if
you
give
them
the
ability
to
opt
out
of
that
notification,
that's
fine.
But
if
you
do
that,
then
everybody's
going
to
still
fall
into
those
two
camps
and
there's
probably
a
completely
different
way
of
workflow
that
the
majority
of
our
developers
tackle.
But
those
are
the
first
two
that
pop
to
mind
for
me.
A
E
E
Like
my
browser,
does
this
notification
on
my
system,
no
matter
which
tab
I'm
on
it'll
say
like
hey,
you
have
a
pipeline
running
now.
I
don't
know
if
we
have
one
for
like
hey
your
build,
there's
a
build
fail
thing.
I
think
that
would
be
a
useful
thing
because
it
only
does
it
if
you
have
that
tab
open
and
if
you
still
care
about
that
pipeline,
I
feel
like.
If
you
don't
want
to
know,
if
you
don't
care,
you'll
go
check
on
it,
then
you
can
just
close
the
tab.
E
B
B
I
think,
like
I
look
at
like
build
level
failure.
Notifications
could
be
like
slack
notifications.
You
know
like
like
they
could
be
super
overwhelming
at
times,
especially
like.
If
I
think
the
git
lab
pipelines
there
are
like
when
master
is
broken
and
we
do
merge
results
pipelines,
you
could
just
be
getting
failure
like
or
like
if
and
I'm
thinking
exceptional
situations,
so
I
shouldn't
be
applying
the
exception
to
the
norm
here.
E
E
B
That
makes
sense:
okay,
yeah
yeah,
I
I
hopefully
our
experiment
and
and
to
be
clear
here
like
james
your
questions,
your
question
you
kicked
us
off
with
is
perfect,
because
I
we
I
don't
really
have
a
I
kind
of-
was
hoping
to
get
started
on
like
what
features
can
we
leverage
from
that?
Your
team
has
already
worked
on
to
so
that
we
can
build
the
tooling
to
do
the
experiment
and
we're
even
diving
further
than
that
into
the
call.
So
this
has
been
great
and
then
looking
ahead
to
provide
feedback
to.
B
B
B
A
E
B
B
Emails
and
those
notifications
were
sent
at
pipeline
end
when
they
were
like
finished
past,
like
everything
was
keyed
on
pipeline
completion,
and
that
was
why
we
geared
that
pi
on
total
pipeline,
like
on
pipeline
completion,
because
most
of
the
developer
feedback
was
what's
pipeline
completion,
but
james
what'd.
You
find.
B
And
that's
where
we
were
talking
about
short-circuiting
it
and
canceling
it
when
we
had
these
failures
before
and
that's
where
we
got
the
overwhelming
feedback
from
a
lot
of
maintainers
who
are
like?
No,
we
don't
want
that,
and
I'm
not
trying
to
put
people
like
I'm
not
trying
to
just
say
maintainers
didn't
want
that,
but
that
was
who
we
solicited
feedback
from,
and
it
was
overwhelming.
A
B
I
I
I
don't
know
I
mean
we
asked
the
question
to
maintainers
because
they're,
like
our
go-to,
for
like
let's
solicit
some
general
feedback,
who
should
we
go
to
first
like
who's
a
group
to
just
so.
We
don't
have
like
a
a
great
way
to
solicit
different
cohort
feedback.
I'll
say
like
a
representative
sample.
A
Workflow
is
that
maybe
fits
for
a
maintainer
differently.
I
guess
I
I
just
I'm
trying
to
tease
out
our
maintainers
doing
that
as
part
of
yep.
I've
approved
this
and
I've
hit
the
merge
button
or
put
it
on
the
merge
train,
and
then
I
don't
want
that
notification
at
that
point
or
as
part
of
day-to-day
day-to-day
development
of
new
features
or
bug.
Pixels.
B
Yeah,
it
would
be
interesting.
The
the
general
feedback
is
when
someone
starts
development
of
an
mr.
They
want
to
know
everything,
that's
wrong
at
that
point
in
time
versus
just
the
first
thing,
that's
wrong
and
just
play
a
progressive
game
of
fix.
The
next
problem,
yeah,
which
I
think
is
more
with.
E
B
E
Okay,
I
was
thinking
this
is
like
a
high
level
thing.
It's
probably
like
too
big
to
tackle
immediately,
but
it'd
be
nice
to
have
like
a
new
status
of
failed
but
still
running
pipeline.
Where
it'd
be
like
a
red
circle
instead
of
the
blue
one,
it's
like
the
this
running
one,
but
it's
red
or
something
like
that.
Be
nice.
B
Yeah
we've
been
teasing
out
this
concept
of
like
a
like
a
deferred
failure,
so
like
foss
broken,
is
like
a
really
good
example
and
maybe
drew
you
may
know
about
this.
But
like
we
let
there's
one.
I
think
actually
right
now
that
I
was
looking
into
right
before
the
call
we'll
let
things
merge
into
the
canonical
repo
that'll
break
foss
gillab
foss
that
we
used
to
prevent,
because
it's
so
infrequent,
but
then
we'll
revert
those
mrs
usually,
if
there's
not
like
a
clear
fix.
B
E
E
E
A
Yeah,
I
think
that's
a
good
so
instead
of
short
circuiting
and
failing
the
entire
pipeline,
you
let
the
rest
of
the
pipeline
run,
but
you
get
the
notification
that
something
has
failed
and
you
can
go
start
to
look
at
that
thing.
First
and
potentially
cancel
the
pipeline
by
pushing
up
effects
on
your
own.
E
C
Yeah,
I
don't
think
a
job
ever
leaves
the
failed
state,
I'm
trying
to
poke
holes
in
why
we
might
like
how
we
might
jump
the
gun
by
doing
a
bunch
of
reporting
while
the
pipeline's
still
running.
But
I
I
don't
think
it's.
I
don't
think,
there's
like
false
positive
risk
in
sending
out
the
notification
early
like
if
you
have
a
failed,
build
your
pipeline
will
fail.
Is
there
any
case
where
that's
not
true.
E
A
B
C
Right
yeah,
I
mean
we
definitely
we'll
have
a
way
to
to
sort
that
out
I
mean
I
don't.
Maybe
we
could.
We
could
get
this
working
for
jobs
that
just
don't
have
retry
configured.
You
know
that's
easy
enough
to
do
and
then
we
can.
You
know
like
there's,
there's
iteration
we
could
take
to
get
there
but
yeah.
I
know
this.
This
sounds
good,
I'm
I'm!
I
don't
think
that
there's
like
a
major
conceptual
blocker
with
like
no.
We
can't
do
this
because
we
don't
know
this
piece
of
information.
Yet.
A
Cool
so
it
sounds
like
we
could
open
up
an
issue
work
with
pipeline
execution
to
try
to
get
a
new
state
for
pipelines
figured
out.
That
is
if
a
job
is
not
allow
failure
or
includes
a
retry.
If
that's
a
thing
change
the
pipeline
status
to
this
new
pending
failure
and
then
the
rest
of
the
notification
should
just
work.
A
C
Then
it
would
be
to
introduce
a
new
pipeline
state,
a
new
pipeline
state,
my
gut
really
big
yeah,
but
I
you're
a
lot
of
people
are
going
to
look
at
you
really
hard.
If
you
try
to
propose
a
new
pipeline
state
is
my
feeling,
so
just
just
a
heads
up
that
if,
if
we
can
use
existing
the
existing
states
of
a
different
reference
object,
we
might
be
able
to
move
on
this
more
quickly.
A
C
I
think
the
state
is
more
at
the
core
of
that
logic,
though,
and
the
notification,
like
our
notification
service
is
more
of
like
a
downstream
feature
from
the
state
change,
and
I
think
a
lot
of
people
are
hooked
into
like
the
particular
state
changes.
So
if
we
can
not
touch
those
but
make
our
integration
to
the
state
change
change,.
B
A
A
Let
me
see
flaky
tests
at
the
project
level
so
that
I
can
proactively
tell
developers
to
go,
fix
them,
or
at
least
research
them,
okay,
cool,
so
kyle
beyond
a
change
in
the
notification
mechanism
or
a
new
state,
whichever
how
we
go
down
that
path.
But
it's
that
notification
happens
to
a
developer
when
a
pipeline
will
fail
where
a
job
has
failed,
build
has
failed,
so
they
can
go
back
and
look
at
it.
A
B
I
so
I
think
what
was
in
the
test
report.
Api
is
what
we
need
for
tooling,
and
that's
what
we
are
going
to
focus
on.
First,
I
would
say
for
the
experiment
in
the
upcoming
release,
so
your
continued
support
and
ideas
on
how
we
can
look
at
the
results
is.
I
would
welcome
that
drew
you
all
always
have
great
ideas
on
on
that
as
well
and
we'll
keep
you
in
the
loop
as
we
continue
to
experiment,
we'll
keep
you
in
the
loop
in
the
issue
I'll
say.
B
And
I
I
could
be
wrong
if
you
have
both
like
again
we're
doing
experiment,
we
could
always
kind
of
do
both
experiments
and
and
look
at
the
results,
the
same
sort
of
way
just
kind
of
due
to
not
do
them
at
the
same
time
do
two
controlled
experiments.
I
should
say,
but
because
that
was
the
other
idea
that
we
talked
about
as
a
team
is,
if
we
don't
have
good
enough
data
on
merger,
the
merge
request.
A
B
Yeah
the
the
challenge
I
see
with
that
is
it.
It
feels
a
little
bit
like
a
like
a
self-fulfilling
prophecy
like
in
my
mind,
the
tests
are
more
likely
to
fail
on
the
default
branch
or
the
most
flaky
tests.
If
we're
running
those
in
the
mrs
of
course,
they're
going
to
be
more
likely
to
fail,
but
they're
also
the
least
likely
ones
that
are
in
the
con
like
in
control.
A
I
think
surfacing
it
will
be
interesting,
though,
and
if
we
can
start
to
get
feedback
from
ep
and
overall
quality
of
how
are
we
tracking
what
we
perceive
to
be
flaky
tests,
because
they're
failing
on
default
or
on
the
main
branch
that
that
would
be
interesting
like?
Are
we
driving
them
down
like
we
would
tech
that
or
measuring
them
in
some
other
way?.
B
B
Our
goal
and
again
we're
kind
of
rescuing
it,
because
we've
had
a
few
between
the
between
the
experiment
that
I
just
talked
about
and
I'll
say
re-looking
at
how
gitlab
bot
is
used
throughout
the
engineering
organization.
Those
two
priorities:
kind
of
getting
increased
in
invisibility
throughout
the
organization.
B
This
may
get
de-prioritized
a
bit,
but
originally
we
were
going
to
have
an
automated
like
the
plan
was
to
have
an
automated
automated
controls
to
kind
of
quarantine
and
de-quarantine
flaky
specs,
based
on
the
amount
of
impact
that
they
were
causing
and
how
frequently
they
are.
I'm
gonna
link
the
kr
right
here.
B
Yeah,
I
don't
think
we're
going
to
end
up
having
the
automated
tooling,
but
the
idea
is
that
we
can
remove
the
noise
that
flaky
specs
cause.
So
then,
when
we
start
looking
at
the
trends
over
the
14
days
in
our
master
branch,
we
start
having
much
clearer
signal
on
what
are
the.
What
are
the
problem?
Specs
that
cause
real
problems.
A
A
A
I
will
so
just
to
sum
up
our
discussion.
I
will
open
an
issue
drew
and
scott
online
and
you
you
folks,
to
help
flesh
that
out
around
notification
mechanism
being
moved
from
pipeline
status
to
build
status.
It
sounds
like
that
might
be
then,
the
next
step
in
kyle's
experiment
after
running
the
previously
failed
tests.
First,
that
was
my
takeaway
and
helping
drive
that
outcome
for
you
all
and
then
we're
going
to
continue
to
work
on
that
project.
A
My
last
agenda
item
here,
if
that
closes
off,
that
discussion
is
wondering
if
we
should
move
this
to
every
other
month.
I
know
we
definitely
haven't
had
much
for
road
map
updates.
Lately.
I
thought
this
was
great
discussion
and
we've
had
some
good
discussions
the
last
couple,
but
we
have
also
had
months
where
we
had
pretty
short
meetings
so
just
wanted
to
check
in
with
kyle
who's
kind
of
our
primary
internal
customer
but
zeph
as
well
on
the
team
and
see.