►
From YouTube: Flake Finder Fridays #000
Description
Rob Kielty and Dan Mangum kick off Flake Finder Fridays, a new Kubernetes community livestream where we explore building, testing, CI and all other aspects of delivering Kubernetes artifacts to end users in a consistent and reliable manner. In this first episode, Rob and Dan are going to look at recent failures in a Kubernetes build job and chat a little bit about why it was failing, what tooling is used to build Kubernetes, and the infrastructure underlying all Kubernetes CI jobs.
A
A
Okay,
sorry
about
that
yeah
I
was
wondering
who's
interrupting
me
and
it
was
me
okay.
So,
let's
start
over
again,
so
my
name
is
robert
kilty.
I
have
worked
as
a
ci
signal
team
lead
and
as
a
ci
signal
shadow
and
as
has
done
what
we
do
on
ci
signal
for
the
kubernetes
project.
Is
we
actively
monitor
the
state
of
ci
as
we
go
through
a
a
release
cycle
for
kubernetes
and
in
doing
that,
work?
B
No,
not
really,
that
was
a
a
great
intro.
One
of
the
things
that
I
did
want
to
add
was
we
were
talking
yesterday
and
rob,
and
I
have
done
quite
a
bit
of
kind
of
flake
hunting
and
fixing
failures
and
that
sort
of
thing,
but
we're
always
continuously
learning
and
one
of
the
things
we
want
to
do
with
this
stream,
is
definitely
encourage
folks
who
are
in
the
chat
or
who
want
to
reach
out
afterwards
in
slack
or
something
like
that
to
definitely
feel
free
to
reach
out
to
us.
B
A
Absolutely
so
I
think
I'm
just
trying
to
think
so.
Should
we
just
head
into
the
issue
that
we're
gonna
look
at
here
we
have.
Am
we
have
a
link
to
a
hackmd
that
I
think
I
shared
on
twitter,
so
I'm
at
rob
guilty
on
twitter.
If
you
want
to
pick
that
up
and
I'm
trying
to
think
what
the
best
way
to
share
that
with
the
audience
is.
B
A
B
So
typically-
and-
and
we
want
to
show
this
as
kind
of
like
the
actual
process
that
we
go
through
and
that
other
folks
go
through
when
they
get
ping
done
on
something
flaking
or
failing,
but
typically
the
ci
signal
team,
which
is
a
release
cycle
team
right.
So
every
release
cycle,
there's
new
ci
signal,
team
members
and
there's
a
team
lead
in
shadows.
B
If
you
want
to
know
specific
things
so,
like
I
said
we're
specifically
looking
at
a
build
job
right,
so
things
that
jobs
that
build
kubernetes
and
then
you
know,
push
up
the
artifacts
for
folks
to
be
able
to
consume,
and
these
are
ci
build
jobs
right.
So
these
are
incremental,
builds,
not
you
know
major
releases
or
something
like
that,
so
the
first
two
that
you'll
notice
are
build
master
and
build
master
fast.
These
are
kind
of
our
two
primary
build
jobs,
but
then
over
on
our
informing
board.
B
Here
we
also
have
this
build
master
canary,
and
this
is
going
to
be
the
subject
of
our
stream
today
and
it's
going
to
talk
about
kind
of
some
of
the
work
that's
been
happening
over
the
last
year
or
so
or
maybe
a
little
longer,
and
it's
also
going
to
touch
on
how
the
release
tooling
connects
to
the
test.
Infrastructure
connects
to
the
kubernetes
code
base
and
more
so
it's
going
to
be
a
fun
one.
B
A
B
Awesome
sounds
good
all
right,
so
the
first
thing
we're
gonna
do
right
is
look
at
the
issue
that
we
were
tagged
in
on
so
joyce
opened
this
issue
and
joyce
is
the
ci
signal
lead
for
the
1.21
cycle
and
they
are
awesome
and
they
are
very
active
at
opening
issues,
and
this
one
was
opened
pretty
quickly
after
this
job
started.
Failing
actually
difference
between
failing
and
flaking
would
be
failing
as
consistent
failures
on
continuous
jobs.
Flaking
would
be
sometimes
failing,
sometimes
passing
so.
B
This
is
a
failing
test
and
joyce
has
denoted
it
as
such
and
you'll
see
that
joyce
went
in
and
got
us
some
information
about
what
was
happening
from
the
build
logs
linked
to
the
issue
and
gave
also
some
some
helpful
context
right
for
us
to
be
able
to
troubleshoot
this
and
for
anyone
else
who
came
along
and
wanted
to
help
out
with
fixing
this
to
do
that.
So
the
ci
signal
team
is
really
great
in
that
regard,
and
this
is
an
awesome
demonstration
of
that
right
here.
B
Can
you
take
a
look
at
this,
which
is
how
sasha
and
I
showed
up
here
and
we
had
some
additional
context
around
some
changes
that
were
recently
made
that
could
have
happened
so
you'll
see
this
kind
of
triaging
here
where
we
assigned
ourselves
and
said
we'll,
take
a
look
at
this
all
right,
so
we
already
took
a
look
at
that
job
that
was
running
and
to
understand
what
the
purpose
of
this
job
is.
It's
important
to
understand
some
context
around
some
changes
that
have
been
made
with
release
engineering
over
the
past
year.
B
So
there's
also
the
test
infra
repo,
which,
if
you're
familiar
with
kubernetes
test
infrastructure,
you've
probably
been
here
before
this
is
where
we
configure
all
of
the
dashboards
to
say
what
tests
should
run,
what
job
should
be
present
on
what
dashboards
etc.
All
of
this
is
a
result
of
the
artifacts
that
are
in
test
infra,
there's
also
some
different
tooling,
as
well
as
images
that
run
jobs
and
kind
of
specify
how
things
can
be
invoked
all
right.
So
once
again,
let's
get
back
to
the
actual
issue
at
hand.
B
Here
there
was
an
error
showing
up
in
the
build
job
and
let's
actually
go
well
actually.
Joyce
gave
us
a
great
link
here
with
a
spyglass
example.
Failure
here,
which
spy
glass
is
kind
of
which
you'll
you'll
frequently
hear,
and
we
mentioned
this
actually
in
the
controvex
talk.
I
remember
you'll
you'll
hear
spyglass
referred
to
as
prowl
you'll
hear
everything
referred
to
as
prowl.
You
can
refer
to
me
as
prow.
A
A
So
that
makes
an
individual
log
line
shareable
when
you're
logging
an
issue
and
that
that's
a
really
useful
feature
that
that
you
kind
of
have
to
hunt
around
and
discover,
and
then
the
final
thing
about
spyglass
is
there's
a
lot
of
runtime
information
at
the
very
bottom
of
the
page
in
relation
to
the
containers
and
the
pods
that
were
spun
up
in
order
to
in
order
to
run
the
job
in
question
and
from
a
troubleshooting
point
of
view.
All
of
that
information
at
the
bottom
is
useful
too.
B
A
B
One
of
them
was
the
actual
dashboard
configuration
was
changed
and
one
of
them
was
the
code
path
that
was
getting
executed,
which
in
this
case
is
just
a
build,
but
a
code
path
nonetheless,
nonetheless
was
also
changed
around
the
same
time.
So
I
want
to
look
at.
Let
me
see
if
I
can
find
this
test
infra
issue
or
this
pr
all
right.
B
So,
as
I
mentioned,
we've
tried
to
move
from
bash
things
or
or
complicated,
hacky
things
to
more
standardized
tooling,
and
that's
and
that's
no,
no
offense
to
anyone
who
has
worked
on
some
of
those
hacky
things
because
they've
gotten
us
to
where
we're
at
today
right
and
it's
a
joint
effort
where
people
are
aligned
and
moving
forward
with
new
tooling.
But
this
is
where
we
moved
this
krell
job
into
the
the
build
canary
job.
B
B
B
We
do
not
support
it.
It's
building
kubernetes,
though
right.
So
it
is
implicitly
supported,
unfortunately,
but
please
do
not
use
it
with
jobs
going
forward,
and
so
what
we
want
to
do
is
move
off
using
things
like
bootstrap.pi,
and
so
we
had
a
canary
job,
which
the
reason
for
this
build
canary
is
another
effort.
B
Did
I
have
this
up
over
here?
Let's
see
build
master
should
be
using
something
like
kubernetes
ci
images,
which
is
the
old
kubernetes
project,
and
then
the
newer
one
here
is
likely
using
something
like
kate,
staging
ci
images
right,
so
we're
moving
over
to
new
infrastructure
to
make
it
more
maintainable
from
a
kubernetes
community
perspective.
B
So
there's
there's
that
aspect
of
it
and
then
there's
also
the
I'm
losing
my
tabs
now,
but
so
there's
that
aspect
of
it,
which
that's
what
the
bootstrap
job
was
serving
its
purpose
of
and
then
there's
also
moving
off
the
bash
right.
So
this
is
a
bootstrap
job
that
was
pushing
to
the
new
infrastructure.
B
So
that
was
a
really
great
thing
and
we
are
excited
to
do
that.
You'll
actually
see
noted
in
here
that
the
no
bootstrap
job,
which
is
the
the
krell
job,
has
been
running
successfully,
but
there
was
also
some
recent
failures
which
we
didn't
think
was
an
issue
at
this
time,
but
it
turned
out
around
the
same
time
as
this
change
was
being
made.
Let's
see
if
I
can
find
the
pr
up
here,
we
switched
to
using
build
x
for
all
of
our
image
builds,
and
I
have
some
links
down
here.
B
B
You
know
what
you're
presented
with
right
now,
so
this
build
master
canary
was
doing
one
thing
and
then
it
was
updated.
But
all
we
see
is
one
continuous
run
right
of
the
same
job,
and
maybe
you
know
we
could
do
things
like
you
know,
use
a
different
name
or
something
like
that.
But
what
you'll
actually
see
if
we
go
back
in
the
failures
is
that
some
of
the
failures
which
we
had
quite
a
few
here,
we're
initially
still
running
the
bootstrap.pi,
I
believe.
B
Well,
actually,
we
had
some
failures
that
were
related
to
infrastructure,
things,
which
this
is
another
good
thing
that
rob
point
out
earlier.
These
are
running
in
pods
in
a
kubernetes
cluster,
so
you
can
have
issues
with
the
infrastructure
itself,
but
anyway,
so
this
started
failing
and
then
was
promoted.
So
then
the
build
master
canary
starts
failing,
and
this
manifested
in
some
interesting
ways
which
we
can
show
an
example
of.
Let
me
actually
just
pull
over
this
slack
conversation
here.
B
So
this
is
pretty
common
with
ci
systems
right
when
you
build
a
new
version,
we
want
to,
you
know,
put
whatever
the
latest
is
so
that
folks
know
what
version
to
pull
down
and
use
if
they
want
the
latest
ci
build.
So
if
we
look
at
this
right
here,
it's
just
showing
the
version
of
the
latest
build
so
that
what
the
issue
that
was
happening
was
this
version.
B
Marker
was
getting
updated,
but
the
ci
images
were
not
available
because
our
job
was
failing
right
in
cube
admin
like
very
good
kubernetes.
Citizens
have
already
changed
over
to
consuming
the
community
infrastructure
right.
So
these
images
and
artifacts
were
available
on
the
google
owned
infrastructure
because
that's
what
our
build
master
was
doing,
but
because
our
canary
job
started
failing
the
the
version
marker
no
longer
had
artifacts
that
were
present
on
the
new
infrastructure,
which
cube
admin
is
rightly
switched
over
to
consuming.
B
So
that's
an
example
of
how
this
affected
users,
even
though
it
was
kind
of
like
intermittent,
builds
right.
These
are
not
actual
releases,
and
I
actually
just
saw
a
question
pop
up
in
here:
do
the
canary
ci
workflows
run
on
prs
also,
or
how
can
we
validate
that
that
the
fix
works
so
we're
going
to
look
at
a
couple
different
ways
how
we
troubleshooted
this?
B
All
right,
so
that
was
promoted,
it
was
failing
and
it
was
confusing
right
because
actually,
the
the
krell
job
and
the
build
master
job,
which
is
using
bootstrap
still
one
of
them,
was
passing
the
build
master
one
and
the
crow
one
was
not
and
they're
running
the
same
exact
commands.
Basically,
even
though
they
have
kind
of
different
infrastructure
around
them,
they
all
end
up
running
in
the
kubernetes
repo
essentially
well.
Actually,
we
can
just
look
at
that
pr
from
sasha.
B
They
all
end
up
running
kind
of
the
same
commands
right,
but
the
the
image
it
runs
in
as
well
as
some
of
the
setup
that
happens
beforehand
is
different.
So
if
we
look
at
the
output
of
the
logs,
let's
see
this
is
a
a
crell
job
here
we
can
actually
take
a
look
at
the
actually,
let's
look
at
the
prow
job,
and
we
can
see
that
the
image
we're
using
is
kate
staging
relinj,
which
stands
for
release
engineering
kate's
ci
builder.
B
A
B
B
Here
we
go
and
if
you
look
at
bootstrap
we're
not
going
to
get
into
all
the
different
things
that
get
set
up
here,
but
basically
these
different
images
set
up
the
ability
to
run
docker
and
docker
right.
So
we
need
docker
present
to
be
able
to
build
these
images
and
push
them,
but
we're
running
within
a
docker
container
as
well
right.
So
the
the
difference
here
is
that
one
of
them
was
failing.
One
of
them
wasn't
right
and
they're
running
the
exact
same
command.
So
how
could
that
be?
Well?
B
The
most
likely
situation
is
that
we
have
different
versions
of
docker
running
right.
Our
environment
has
to
be
different
somehow
so
that
specific
flag
with
the
dash
dash
platform
not
being
recognized
that
has
to
do
with
how
build
x
works,
and
we
can
actually
take
these
different
images
right
and
we
can
run
them
locally
and
kind
of
see
what's
inside
of
them,
there's
other
ways
to
troubleshoot
as
well.
I
particularly
like
this
because
it
allows
me
to
play
around
in
it
a
little
bit.
B
So
this
is
the
bootstrap
image
that
I
mentioned,
and
I'm
gonna
run
this
locally
and
let's
just
check
out
kind
of
what's
inside
of
here,
all
right,
so
we're
inside
and
let's
see
what
docker
version
we
have
all
right.
We
have
docker
version
20.10.2
all
right.
Let's
now
take
a
look
at
the
ci
builder
image
and
check.
What's
inside
of
there
docker
version
all
right.
We
have
19.0
1903
13.,
so
obviously
an
older
version
right
so
right
away.
B
So
it's
not
a
real
platform,
but
we're
going
to
see
the
same
exact
issue
that
we're
seeing
in
the
jobs
right
we're
seeing
this
unknown
flag
platform
which
to
me
this
is
a
bit
of
a
uninformative
error
because
what's
really
happening
here,
is
it's
not
recognizing
build
x,
but
the
the
main
error?
It's
giving
us
is
that
it
doesn't
recognize
this
platform
flag
if
we
actually
ran
it
without
platform.
I
think
docker
build
x,
yeah
build
x
is
not
a
docker
command,
so
that
would
have
been
maybe
a
little
more
helpful.
B
But
essentially
buildex
is
a
newer
part
of
the
docker
cli.
It's
basically
a
different
engine
that
you
can
use,
and
so
we
don't
have
that
enabled
by
default
and
how
you
can
enable
that
by
default
or
not
by
default,
but
how
you
can
enable
that
is
by
setting
docker,
cli
experimental
to
enabled-
and
let's
see
here
if
we
actually
have
that
set
and
if
you're
watching
from
home
you're,
probably
like
no
dan,
we
obviously
don't
have
that
set,
or
else
this
would
be
working
and
you're
right.
B
We
don't
and
in
the
other
docker
version
right,
docker
cli
is,
is
no
or
build
x
is
no
longer
experimental,
so
it
was
working
by
default.
So
when
we
went
ahead
and
switched
over
to
using
buildex
in
one
environment,
it
was
saying
okay,
I
recognize
build
x
and
platform
and
all
the
things
that
go
with
it,
and
so
that's
going
to
work
and
in
this
other
one
it's
not
so
the
there's
a
number
of
different
solutions.
B
B
You
know
older
versions
of
docker.
We
would
like
for
this
to
work
out
of
the
box
and
there's
some
build
x
setup
that
I'm
not
going
into
that
could
also
make
this
complicated,
but
this
was
the
kind
of
the
the
best
solution
for
the
greatest
number
of
community
members,
and
it
was
a
short-term
fix
right.
So
we
could
get
that
in.
We
actually
didn't
have
to
build
new
images
or
anything
like
that,
so
that
was
nice
and
we
thought
all
right
good.
We
solved
the
problem.
B
But
let's
look
at
a
later
failure,
particularly
the
one
right
before
it,
it
flipped
to
passing
and
see
what
we're
getting
now
inside
of
here,
all
right,
so
we're
building
all
of
our
go
targets.
Okay,
so
that
looks
good
before
what
was
happening
is
we
were
having
an
issue
with
docker
build?
Okay,
it
looks
like
that.
B
Our
docker
builds
are
running
here,
at
least
some
of
them,
and
this
this
is
kind
of
the
the
output
you
get
when
you're
running
build
x,
but
we
see
that
there
are
some
issues
here
with
quiet
currently
not
implemented.
That
kind
of
just
sounds
like
a
warning
to
me,
but
you
know
could
be
something:
sasha
noticed
it
as
well
and
sasha
said
we
can
clean
that
up
and
also
we
want
the
build
logs
to
be
verbose
if
possible.
B
So
while
sasha
was
kind
of
working
on
this,
I
was
taking
a
look
and
trying
to
fix
it
and
we're
like.
Oh,
this
will
probably
help
right
we're
going
to
get
more
verbose
build
logs
here.
Maybe
dan
will
be
able
to
fix
it
and
we'll
be
able
to
move
forward,
but
we
didn't
really
think
this
would
have
a
big
impact
on
it.
So
this
got
merged,
and
this
was
a
few
days
later,
as
you
can
see,
there's
lots
of
failures
and
all
of
a
sudden
it
turned
green.
B
B
We
can
see
that
the
quiet
currently
not
implemented
is
no
longer
present
there
right
and
what
was
happening
was
and
once
again
I
want
to
reiterate
that
so
we
added
the
docker
cli
experimental,
which
should
have
solved
the
difference
between
the
bootstrap
image
and
the
kate
ci
builder
image,
but
it
still
had
they
both
had
the
quiet
flag
right.
So
why
was
the
the
build
master
job
still
passing?
B
Well,
I
ran
this
locally
with
my
own
docker
installed,
which
I
believe
I
have
1903,
probably
which
a
lot
of
folks
are
still
using,
and
I
also
got
the
error.
So,
let's
just
try
that
with
docker
build
x,
build
there's
nothing
actually
here,
but
quiet
currently
not
implemented,
and
you
see
it
didn't
even
try
to
build.
And
if
I
look
at
what
the
output
is,
we
get
exit
1
right,
so
that
is
causing
an
issue
because
in
our
release
script
over
here.
B
And
actually
I'll
do
this
in
the
terminal,
because
rob
gave
me
some
good
feedback
about
that.
It's
a
little
bit
easier
to
follow
in
an
editor,
so
I'll
try
to
do
that
and
if
folks,
viewing
also
have
more
feedback
around,
you
know
how
this
can
be
easier
to
follow
along
definitely
give
us
that
that
feedback
as
well.
But
basically
this
is
the
the
script
that
sasha
was
modifying
right
with
the
build
x
commands
and
then
I
later
added
the
cli
experimental
flag.
A
B
B
Enabled
docker
build
x
version
all
right,
so
we
see
that
we
also
have
an
older
version
of
docker
buildex
in
this
image,
and
so-
and
I
I
think
I
actually
did
this
prior
to
doing
this.
Little
exercise
wanted
to
search
for
the
issue
and
see
if
we
could
find
where
anyone
else
was
seeing
this
quiet
not
implemented,
and
here
is
the
pr
to
build
x.
B
B
So
it
was
just
warning
about
that
quiet,
flag
and
in
the
kci
builder
job
we
were
actually
erroring
on
that
which
was
causing
our
job
to
fail,
was
causing
folks
to
not
be
able
to
consume
from
the
new
community
infrastructure
and
actually
the
way.
I
think
that
we
ended
up
catching.
This
was
that
sasha
had
a
newer
version
of
buildex
on
his
local
machine,
so
it
was
actually
kind
of
happenstance
that
it
was
caught,
but
we
were
chatting
about
it
in
slack
and
that's
how
we
got
to
it
all.
A
Right
so
this
is
kind
of
yeah
there's
a
kind
of
a
whole
inception
vibe,
as
is
often
the
case
in
in
container
runtime
stacks.
I
suppose,
if
you
were
coming
to
this
cold,
this
would
be
quite
intimidating,
be
fair
to
say,
you're,
not
coming
one
of
the
things
or
even,
if
you're
not
going
to
a
cold
yeah,
but
but
in
terms
of
in
terms
of
in
terms
of
scaling
in
terms
of
scaling
the
wall
yeah,
it's
not
a
learning
curve.
A
It
is
a
climbing
wall
with
an
overhang,
and
you
know
a
couple
of
holes
taken
out.
It
can
seem
like
that,
but
but
it's
important
to
note
that
that
that,
as
a
team
and
as
a
community
that
there's
a
lot
of
leaning
on
each
other
in
terms
of
in
terms
of
getting
to
the
root
cause
of
of
these
issues
would
be
fair
to
say.
I
think.
B
Absolutely
absolutely
and
and
lots
of
that
happens
in
the
slack
channel,
so
it
can
be
really
helpful
just
to
jump
in
there
and
ask
questions
and
that
sort
of
thing
I
know
when
I
was
looking
through
this
failure
here
I
was
asking
stephen
questions.
I
was
asking
ben
questions
and
they're
all
providing
really
helpful
context,
including
looking
at
how
we
enabled
build
x
in
k,
release,
which
is
with
a
script
that
we
actually
borrowed
from
kind
which
obviously
ben
does
a
lot
of
work
on.
B
Go
ahead
and
note
this
real
quickly,
but
one
of
the
things
that
happened
in
the
meantime
is
steven
noticed
that
those
cube
adm
folks
were
having
trouble
consuming
the
latest.
So
he
went
in
and
went
ahead
and
re-enabled
that
no
bootstrap
or
that
bootstrap
job
excuse
me
and
just
named
it
build
kate's
infra,
and
this
allowed
us
to
get
those
images
available
again
right
until
we
could
determine
what
the
issue
was
and
provide
a
fix.
A
So
one
of
the
things
that
I
was
going
to
say
down
just
if
you're,
if
you're
coming
to
this
completely
cold,
that
that
the
kubernetes
as
an
application
resides
in
the
kubernetes
kubernetes
repo
on
github,
the
we
have
a
separate
intro.
We
have
a
separate
repo
called
test
infra
and
that
and
contains
all
of
the
tooling
that
supports
and
underpins
ci
in
the
kubernetes
project
and
then,
as
a
separate
as
another
repo.
A
We
have
sig
release
and
and
a
lot
of
the
tooling
that
we've
been
talking
about
today
is
resides
there
and
is
tracked
there
and
that
tooling
is
used
to,
I
suppose,
build
and
build
kubernetes
releases
to
be
released
to
the
the
public
and
and
one
of
the.
If
you're
coming
to
the
project.
I
knew
one
of
the
things
that
you
kind
of
have
to
sort
of
feel
your
way
around
and
learn
is,
is
what
repos
do?
B
Absolutely
so
yeah,
that's
that
is
most
of
the
the
failure
we
actually
wanted
to
show
today.
I
don't
see
any
major
questions
here
that
I
don't
think
that
we
haven't
covered
in
the
chat,
but
I
know
we've
been
going
for
40
minutes.
We
have
about
10
more
minutes
that
we
could
fill
here.
Rob
was
there
anything
you
wanted
to
specifically
look
deeper
into
that.
I
was
walking
through
there.
A
I'm
trying
to
think
like
I
mean
I
think,
for
somebody
to
go
through
it
all
those
links
are
there.
The
main
thing
I
just
want
to
reiterate
is
is
that
everybody
doesn't
know
everything
and
that
if
you
keep
up,
if
you
keep
on
asking
questions
and
tackling
issues
and
ask
those
questions
in
in
the
slack
channels
and
that
that
you'll
get
the
support
you
need
and
any
contribution
that
you
want
to
make
will
be,
will
be.
Welcome.