►
From YouTube: OKD Prow Build Process Debugging Demystified Michael McCune, Denis Moiseev & Diane Mueller - OKD WG
Description
In this video, you will learn about the process for examining OKD release artifacts to discover when there are failures in the release testing. You will see how to interpret the releases page, how to find the Prow interface to the continuous integration testing infrastructure, and how to investigate the test results to unlock the deeper mysteries of OKD debugging. You will also learn about online tools for researching the testing workflows, for searching log messages to find related failures in other tests, and for reporting the bugs and issues you have found.
Guest Speakers: Michael McCune, Denis Moiseev, and Diane Mueller
For more information, please visit: https://okd.io
Addtional Resources & Links:
K8s prow arch:
https://docs.prow.k8s.io/docs/overview/architecture/
OCP CI docs home:
https://docs.ci.openshift.org/docs/
https://docs.ci.openshift.org/docs/getting-started/useful-links/ - useful links to various services. Links to talks and presentations:
https://docs.ci.openshift.org/docs/how-tos/artifacts/ artifacts explanation
in video, started here
https://www.okd.io/installation/
went to here for releases
https://amd64.origin.releases.ci.openshift.org/
place to find prow configurations
https://github.com/openshift/release
Ci step registry, useful for discovering how test jobs flow together
https://steps.ci.openshift.org/
CI log search, useful for finding similar test failures
https://search.ci.openshift.org/
place to log bugs
https://issues.redhat.com
please use project “OCPBUGS”
A
Welcome
everyone
today
is
friday,
the
2nd
of
september
2022-
and
this
is
the
okd
debug
session,
where
we're
going
to
dive
into
how
you
can
look
at
open
shift,
releases
and
kind
of
understand.
What's
going
on
with
them
to
begin
with,
I'm
michael
mccune
and
I
work
for
red
hat
as
a
software
engineer
on
the
cloud
infrastructure
team
working
on
openshift
and
I'm
joined
today
by
dennis
moy
sieve
and
diane
mueller.
So
dennis.
Why
don't
you
introduce
yourself.
C
Cool
well
thanks
dennis
for
joining
us
today
and
I'm
diane
mueller,
I'm
one
of
the
co-chairs
of
the
okd
working
group
and
we're
going
to
start
doing
a
series
of
talks
like
this
one,
and
we
will
go
deeper
on
each
of
these
topics
that
we
cover
today.
But
we
thought
this
was
a
good
place
to
start
so
mike
drive
through
this
debugging
session
for
us.
Thank
you
very
much.
A
All
right
sounds
good,
so
I'm
gonna
follow
a
path
here
that
probably
many
okd
users
follow
where
you
will
start
by
looking
for
how
do
I
install
okd
and
as
I'm
looking
for
that
information,
how
can
I
find
out
which
versions
of
okd
are
good,
which
versions
have
not
passed
the
testing
and
really?
How
can
I
get
deeper
into
understanding
what's
going
on
there
now?
A
So
we'll
start
to
look
at
that.
So
if
I
was
an
okd
user-
and
I
am
one
when
I
go
to
do
an
installation-
oftentimes
I'll-
come
to
the
okd
dot
io
website
and
look
at
the
installation
page
now
as
you're
getting
started,
one
of
the
things
we
talk
about
is
where
you
can
find
releases
from
and
the
main
place
to
go
for.
This
is
the
ci
release
infrastructure.
So
we'll
click
on
that
and
what
we
see
here
is
a
release
status
page,
and
this
is
showing
us.
A
The
output
of
all
the
release,
builds
that
have
happened
and
a
release
build,
creates
the
release
artifacts
that
we
end
up
using
when
we
want
to
deploy
okd,
and
so
you
can
see
that
you
know
the
releases
are
sorted
here
by
the
version
of
okd.
They
are
as
well
as
the
date
that
they
were
produced
and
then
there's
a
it
talks
about
a
phase
whether
it
was
accepted
or
rejected,
and
this
this
is
determined
based
on
if
it's
passing
the
test,
if
it's
passing
the
conformance.
A
So
if,
during
the
release
build
it
does
not
pass
all
the
ci
tests,
then
we
would
start
to
want
to
understand.
Why
was
this
one
rejected
and
so
we'll
drill
a
little
deeper
now
and
understand,
what's
happening
here?
So
maybe
we'll
check
out
this
rejected
one,
and
we
can
see
the
status
of
what
happened
with
this
build
at
the
beginning.
It
gives
us
some
instructions
on
how
we
could
you
know,
get
this
version
of
the
image
and
extract
it,
and
then
it
tells
us
where
the
failures
are
starting
to
happen.
A
Now
you
can
see
that
even
in
this
test,
it's
giving
us
information
about
other
releases
that
have
succeeded
in
some
of
the
tests.
We
can
see
that
marked
here
where
there's
many
tests
happening.
We
can
also
see
there
are
several
failures
here
now.
The
question
is:
how
do
we
start
to
investigate
these
failures
and
okd,
much
like
openshift
and
and
kubernetes
uses
prow
as
the
primary
infrastructure
for
running
its
testing?
A
So
what
we
can
do
is
click
on
one
of
these
screens
and
what
we'll
see
is
the
prow
screen
that
tells
us
about
what
failure
happened
here
now.
In
this
case,
this
failure
looks
pretty
interesting,
but
let's
talk
about
the
various
pieces
of
what
we're
looking
at
here
is
prowl,
with
a
little
bit
of
customization
for
openshift
and
it
and
at
the
top
we
can
see
the
test
that
was
running
when
this
happened.
So
this
was
release.
A
A
A
This
test
is
maybe
there's
some
sort
of
platform
problem
happening,
so
I'll
go
back
to
the
prow
screen
now
and
one
of
the
other
links
here
we
can
look
at
the
artifacts
and
these
are
the
artifacts
that
are
created
with
every
job
run.
When
we
look
here,
we're
looking
at
pretty
much
like
an
ftp
list,
where
we
can
see
various
pieces
that
have
come
out
of
the
the
job.
So,
as
you
start
to
understand
the
ci
testing
infrastructure
more
and
as
you
start
to
look
at
specific
tests,
every
test
might
have
different
artifacts.
A
It
creates,
and
so
we
look
in
the
artifacts
directory
you
can
see.
This
test
doesn't
have
a
very
deep
amount
of
data
here,
but
it
has
something
about
the
build
resources
and
the
steps
that
took
place
and
similarly,
there
are
files
here
that
talk
about
kind
of
how
the
job
was
set
up
and
the
history
and
everything
some
of
these
are
exposed
on
the
prowl
page,
but
other
of
them
you'll
want
to
come
here
for
like
in
some
cases,
tests
might
have
a
must
gather
embedded
within
the
artifacts.
A
A
And
if
I
click
on
this,
you
can
see
that
there
was
something
happened
here
could
not
resolve
source
image
stream,
so
this
might
be
more
of
a
flake.
This
might
be
very
difficult
to
investigate
because
it
looks
like
there
was
some
problem
in
the
setup,
and
you
know
the
image
stream
didn't
get
didn't
get
found
here,
but
we
could
investigate
further
by
going
into
the
build
log,
and
this
build
log
is
really
like
a
full
examination
of
everything
that
happened.
You
can
see
that
it
talks
about
here.
A
A
B
So,
for
me
that
looks
like
an
underlying
infrastructure
problem
or
something
like
an
issue
with
the
cluster.
Where
we
run
in
our
our
test
runs
seems
the
image
streams
was
not
synchronized
for
some
reason
and
yeah
pretty
much
everything
always
in
that
kind
of
situations.
I
just
go
and
asking
people
who
are
in
charge
for
our
tested
infrastructure
to
take
a
look
on
this.
A
Right,
that's
pretty
much
what
I
think,
too,
and
so
this
this
kind
of
brings
us
into
talking
about
you
know
flakiness
and
flaky
tests,
and
so
when
you're
looking
at
a
test.
So
when
something
like
this
happens,
there's
not
a
lot.
You
can
do
with
the
openshift
code
to
fix
it
immediately.
This
test
might
require
interaction
with
the
folks
who
are
maintaining
the
ci
infrastructure
or
it
might
require
updates
to
the
ci
infrastructure.
So
this
might
be
a
really
early
warning
that
well
this.
A
This
test
didn't
actually
fail,
but
something
deeper
is
happening
on
the
infrastructure.
So
let's
go
back
and
take
another
look
at
the
tests
here
and
see
if
we
can
find
a
test
failure
that
maybe
looks
more
interesting
for
what
we're
looking
at
so
I'll
click
on
another
failed
release
here
all
right.
This
looks
very
similar
to
the
previous
one.
A
So,
let's
go
to
the
previous
version
when
we
know
something
succeeded
here,
but
let's
see,
if
maybe
there's
a
little
different
output
in
the
testing.
Okay,
we're
seeing
the
similar
problem
here
so
there
might
have
been.
There
might
have
been
a
deeper
issue
going
on
all
right,
so
this
test
failure
starts
to
look
a
little
bit
different.
A
We
see
that
at
the
top
here,
there's
these
new
categories
for
debug
tools
and
e
to
e
chart,
and
these
will
give
you
information
kind
of
deeper
about
what's
happening
with
the
build
and
I'll
just
I'll
look
through
them
quickly,
but
some
of
these
tools
can
be
very
overloading
because
it's
a
lot
of
information.
So
generally,
I
don't
spend
a
lot
of
time
here.
I
don't
know
dennis.
Do
you
spend
a
lot
of
time
looking
at
looking
at
the
various
pieces
of
the
debug
tools
and
whatnot.
B
Yeah,
so
there
is
a
quite
nice
debug
tool
which
the
which
calls
premier
clues
the
last
thing
here
so
and
that's
basically
a
thing
which
extracts
a
metrics
and
parameters
dump
from
from
cluster
which
we
was
tested
there
and
that
helps
quite
a
lot
to
debug
and
analyze.
Some
kind
of
performance
issues
and
yeah.
That
kind
of
things.
A
That's
a
good
call
out
on
promecleus,
let's,
let's
take
a
look
at
the
promecus
page
and
see
what
it's
saying
right.
Oh
it
seems
like
we
failed
to
find
a
metrics
arc
all
right
perhaps,
and
perhaps
we
should
come
back
to
promiscuous
then
in
short,
though
promiscuous
is
a
tool
that
would
allow
us
to
to
look
through
and
examine
metrics
that
happen
on
on
test
runs.
But
we'll
come
back
to
this.
A
B
So,
quite
rarely
these
I
using
in
situations
when
we
have
a
performance
degradation
within
our
tests.
So
probably
that's
our
team
specific,
but
this
chart
I'm
using
when
we
have
like
lowness
and
performance
degradation
when
machines
created
creating
and
that
kind
of
things.
So
this
chart
has
basically
shown
you
how
much
time
each
test
was
taken.
A
A
If
we
keep
going
a
little
bit
here,
we
see
the
j
unit
section
like
we
had
seen
before,
but
now
we
see
some
different
information
here
we
see
some,
you
know,
failure
to
launch
this
ede
test
container
and
we
also
see
these
sig
networks
dig
node
and
then,
and
then
these
flaky
tests,
so
we'll
get
into
a
little
description
here.
What
these
are
each
one
of
these
is
a
different
test
that
is
being
run
on
the
cluster
and
some
of
them.
You
can
see
this
one
says:
sig
network
pods
should
successfully
create
sandboxes
by
other.
A
A
One
of
these
debug
lines
right
and
you
say,
okay,
I
start
to
look-
and
I
can
see
there's
artifacts
here
for
my
cluster
and
I
realize
this
highlighting
is
not
working
very
well,
but
you
can
see
it's
calling
out
specific
nodes
and
it
says
this
is
never
deleted.
A
reason
failed
create
pod
sandbox.
You
know
I'm
not
familiar
with
these
tests
in
specific,
but
this
is
what
I
would
start
using
to
start
looking
back
through
the
code
and
debugging
what
was
happening.
A
A
So
as
we
go
down
the
I
might
investigate
more
tests.
You
can
see
this
one
looks
a
little
more
looks
a
little
more,
a
little
less
verbose
and
is
just
telling
us
something
about
the
testing
there.
So
we
would
probably
need
to
look
at
that
test
to
figure
it
out
dennis.
Do
you
have
any
comments
about
like
looking
at
how
you
go
about
looking
at
these
at
these
test
failures.
B
Why
that
test
was
failed?
First
of
all,
I'm
trying
to
locate
the
exact
log
messages
which
might
indicate
on
the
reason
why
this
test
was
failed.
Was
that
some
sort
of
time
out
and
maybe
try
to
find
some
breadcrumbs
about
about
the
entity
like
a
note
in
this
case,
or
maybe
services
or
machines
or
that
kind
of
stuff.
A
A
If
we
know
that
the
underlying
infrastructure
might
not
necessarily
always
do
what
we
expect
it
to
or
if
there
are
some
sort
of
miss
in
in
a
creation
event
or
something
we
know,
these
tests
could
be
flaky
and
so
oftentimes,
it's
important
for
us
to
kind
of
separate
out
the
flaky
tests
first,
because
if
we
spend
a
lot
of
time
debugging
this,
we
might
just
find
out
that
it
was
a.
It
happened
to
be
a
problem
with
the
underlying
infrastructure.
A
You
know,
I'm
sure,
we've
all
had
experiences
where
we
go
to
click
an
instance
or
we
create
an
instance
in
some
infrastructure
and
for
whatever
reason
it
doesn't
happen.
Some
service
internally
failed
and
we
don't
get
a
good
instance
out
of
it
and
this
kind
of
represents
where
the
flakiness
comes
from
and
if
we,
if
we
scroll
down
a
little
forward
a
little
further,
you
know
again
we
come
to
the
build
log
and
in
this
case
you
can
see
because
many
more
tests
ran
the
build.
Log
is
quite
long
here
and
again.
A
So
you
can
see
that
there's
a
lot
of
testing
that
just
happens
that
emits
logs
that
we
don't
need
to
save,
and
so
you
you
could
go
back
and
look
for
those
if
you
wanted
to,
but
there's
really
usually
not
a
lot
of
reason
to
do
that,
but
it
does
show
us
the
failures
that
were
enumerated
in
the
top
part
of
this
and
gives
us
a
little
more
information
on
the
out
around.
You
know
what
was
happening
when
that
test
failed.
A
So
we
might
want
to
look
back
in
here
to
see
if
there
is
other
information
about
where
that
test
came
from,
and
in
this
case
it
doesn't
look
like
there's
a
lot
of
information
that
might
help
us
here.
You
know
it's
talking
about
other
failures
or
other
tests
that
happen
during
the
same
time
that
might
have
and
then
here's
another
test
failure.
A
You
know
that
we
we
got
captured
above
and
again,
you
might
need
to
expand
these,
so
you
could
see
the
context
around
the
failure
as
it
occurred,
so
we'll
keep
going
I'll
skip
past
some
of
this,
because
this
is
you
know,
this
is
all
information
that
is
that's
good
to
investigate,
but
may
not
necessarily
be
what
we're
looking
for.
So
I'm
going
to
scroll
by
kind
of
quickly
here
just
so,
we
can
show
the
bottom
of
this
page
now
at
the
bottom
of
this
page,
we
see
like
information
about
the
job
pod.
A
A
You
know
a
little
bit
of
information
about
what
how
the
pod
was
running
on
the
ci
infrastructure,
and
this
might
be
useful
if
you
were
actually
trying
to
debug
the
ci
infrastructure,
or
you
thought
there
was
a
problem
with
the
infrastructure
that
wasn't
specific
to
the
test.
So
one
more
thing
I'd
like
to
look
at
here
before
we
before
we
go
to
the
next
part,
and
that
is
the
artifacts
section.
A
So
again,
this
test
ran
much
longer
than
the
other
ones.
So
what
we'd
like
to
do
is
look
at
the
artifacts
here
and
just
see
if
we
can
see
you
know
what
more
is
here.
So
we
see
there's
more
information
here
than
the
last
one.
There's
the
same
build
resources,
but
we
have
some
of
these
other
directories
as
well,
and
I
know
from
experience
that
if
we
start
to
look
in
release
and
artifacts,
you
know
sometimes
we
might
find
a
must
gather
in
here.
A
These
will
be
named
based
on
the
tests
and
so
in
this
launch
test
you
can
see
there
are
several
steps
that
were
happening
here.
These
were
all
the
individual
steps
of
the
workflow
that
took
place
and
one
of
them
was.
This
gather
must
gather,
which
is
a
step
that
we
run
at
the
end
of
our
ci,
so
that
we
can
gather
artifacts
from
it.
So
if
we
look
in
the
artifacts,
you
can
see
them
something.
A
There
is
a
tar
for
the
must
gather
here
that
you
could
download,
and
then
there
are
some
of
the
other
artifacts
that
will
be
bundled
inside
of
that.
So,
for
example,
the
event
filter
is
an
html
file
inside
the
mustgather
that
allows
you
to
look
at
the
events
that
happen
within
that
cluster,
and
I
realize
this
is
quite
small,
but
this
is
another
thing
that
you
can
use
to
start
examining
how
a
failure
happened
within
the
cluster,
and
these
must-gathers
will
be
very
important
in
terms
of
debugging.
A
B
Yes,
so
if
you
could
go
back
to
that,
folder
where
you
found
my
mask,
so
there
is
another
folder
here
on
this
screen,
which
goes
gather
extra
so
and
that
gather
extra
is
basically
unpacked
mud
gather
archive
with
a
bunch
of
additional
things,
and
there
is
like
oc
cmds,
which
is
quite
useful
for
debugging
cluster,
because
you
can
take
a
look
on
various
oc
command,
outputs
such
as
cluster
operators,
node
versions
and
that
kind
of
stuff.
So
I
use
that
quite
a
lot,
at
least
for
defining
the
version
of
the
cluster.
B
The
cluster
current
states
of
how
it
looks
like
also
there
is
a
dump
of
bot,
manifests,
namespaces,
manifests
and
another
cluster
resources,
so
which
is
extremely
useful,
sometimes
and
basically
using
that.
I
do
not
have
any
necessity
to
unpack
that
entire
mass
gather
and
dig
in
yaml
files
there.
I
could
use
just
a
web
interface,
so
another
useful
thing,
especially
during
debugging
infrastructures.
B
It's
on
level
above,
I
guess,
and
there
is
gather
audit
locks
and
gather
gcp
console
so
audit
logs.
It's
a
logs
of
an
api
server.
So
and
here
we
could
find
communication
between
wireless
openshift
components
and
api
servers.
So
from
these
logs
we
can
get
an
idea
what
components
goes
to
api
server
change,
what
resources
and
so
on.
B
Speaking
about
another
folders
here
like
api
config,
api
provision,
d
provision
and
that
kind
of
stuff.
So
that's
connected
to
the
installation
process.
So
and
here
we
can
find
install
locks,
release
image
version,
information,
information
about
how
cluster
was
destroyed
and
so
on
which
sometimes
quite
useful,
especially
when
you
have
to
debug
installation
and
that
kind
of
stuff.
B
So
that's
actually
a
log
log
file
of
a
program
which
which
was
engaged
for
running
that
that
step
so,
for
example,
installations
type
installs,
the
cluster
e2e,
your
test
steps
perform
the
tests
and
so
on,
and
there
is
just
like
program,
some
sort
of
observer,
which
actually
runs
that
tests
binary
or
installation
program,
or
that
kind
of
things
so
and
when
you
cannot
find
any
information
in
respective
folders.
So
this
file
would
be
extremely
useful
for
reporting
problems
back
or
for
just
getting
an
idea.
What
was
going
on
there.
A
A
I'm
gonna
have
to
engage
hyper
warp,
speed
all
right
so
just
to
cover
what
we
looked
at
here.
We
looked
at
the
release
status,
page
we've
drilled
into
the
individual
releases,
and
then
we've
looked
at
the
prowl
for
failed
and
successful
jobs.
So
we've
looked
over
prowl
we've
seen
like
you
know
the
tests
that
come
out
of
it,
the
test
that
it's
running
so
the
next
question
might
be.
How
is
proud
configured?
How
do
these
various
tests
run
and
and
what
are
we
looking
at
here?
A
Now
prow
has
a
very
deep
configuration
and
I'm
certainly
not
going
to
be
able
to
cover
it
all
here,
but
what
we
can
do
is
start
to
show
where
these
things
are
originating
from
and
give
you
some
information
about
where
you
can
find
out
how
to
understand
that.
So,
for
the
time
being,
the
information
about
our
prow
is
contained
in
the
open
shift
release
repository,
and
I
I
have
a
feeling
that
okd
might
change
this
at
some
point.
A
So
when
you
start
to
take
apart
one
of
these
tests
to
understand
how
it
was
configured,
you
might
first
look
at
the
configuration
and
this
lists
kind
of
where
these
projects
are
in
in
respect
to
how
they're
organized
in
github.
So
you
might
start
to
look
and
say
I
want
to
find
the
open
shift
test
and
we
saw
that
origin
was
a
part
of
the
the
test
pattern
that
we
looked
at
before
and
then
the
open
shift
origin
repository
is
where
we
hold
a
lot
of
the
end
to
end
test
from
upstream.
A
So,
as
I
continue
to
scroll
here,
you
can
see
there's
a
lot
of
different
tests
that
are
contained
here.
What
I
would
like
to
find
is
our
origin
test
or
the
configurations,
and
so
now
you
can
see
the
configurations
here
and-
and
you
see,
these
are
tagged
and
I'll
increase
the
screen
size
just
to
make
this
a
little
easier
to
read,
they're
tagged
based
on
you
know,
kind
of
where
they're
coming
from
and
then
what
version
is
being
is
being
set
up.
A
A
A
Yeah,
so
this
is
the
source
of
truth.
It
may
not
be
the
most
user
friendly
way
to
look
at
it,
but
if
you're
the
type
of
person
who
wants
to
look
at
the
configuration
files
and
understand
how
these
things
are
set
up,
this
is
the
place
to
come,
and
you
can
see
that
as
we're
scrolling
through
here.
We
see
a
bunch
of
these
things
as
ede
aws
jenkins,
as
ede
gcp
image
ecosystem.
A
These
are
each
different,
different
tests
that
we'd
like
to
run
and
we're
going
to
call
it
this,
and
then
you
can
see
there's
a
little
explanation
for
when
this
should
run,
and
it
gives
some
some
kind
of
a
filter
here
to
detect,
and
that
says
what
steps
should
be
run
now.
This
cluster
profile
tells
what
kind
of
cluster
it
should
be
run
on.
A
So
like
we
know
we,
there
has
to
be
a
cluster
for
the
test
to
run
on,
and
this
is
what
it
refers
to
and
then
this
workflow
is
what
tests
it
should
actually
run.
And
so,
let's,
let's
take
this
open
shift,
ede
gcp
image,
ecosystem
test
and
I'm
not
quite
familiar
with
this,
but
you
know
we
can
just
use
it
as
an
example
and
we
should
be
able
to
go
and
look
in
the
in
the
configuration
now
and
see
or
in
the
jobs
now
and
see
this
listed
as
a
job.
A
And
this
may
be
difficult
to
find
from
here,
so
we'll
show
another
way
to
look
for
these
things
using
a
search,
but
we
could
look
in
the
jobs
then,
and
you
can
see
the
the
directory
structure
is
very
similar
here.
So,
let's
see
if
we
can
find
the
same
thing,
I
guess
origin
does
not
exist
the
same
way
here,
so
this
might
be
coming
directly
out
of
the
step
register
and
and
dennis
feel
free
to
correct
me
here.
If
I'm,
if
I'm
going
off
in
the
wrong
direction,.
A
All
right,
so
what
I'm
going
to
do
is
I'm
going
to
search
for
this
in
this
repository,
and
what
we
should
see
is
that
it's
used
in
a
bunch
of
places,
so
you
can
see
that
it's
used
as
the
workflow
in
several
different
tests,
and
I
realize
I
skipped
by
the
interesting
one
right
away,
but
just
to
show
how
this
piece
of
information
could
be
used
over
and
over
again.
But
we
can
see
that
in
the
ci
operator,
step
registry
open
shift
e
to
e
gc
image
ecosystem,
which
is
the
way
it's
named
here.
A
A
Then
it
wants
to
run
the
openshift
ede
test
and
then
afterwards
it
will
run
the
ipi
gcp
post,
and
so,
if
we
start
to
look
into
each
of
these
steps,
we'll
start
to
see
what
it's
running
for
each
one
of
those
and
presumably
in
the
ipi
gcp
post,
is
where
it
will
do
the
cleanup
and
do
the
must
gathers
and
those
type
of
things
and
then
the
openshift
ed
test
is
where
it
runs,
actually
all
the
tests.
So
that's
that's,
probably
the
most
interesting
one.
A
So,
let's,
let's
continue
our
exploration
here
and
we'll
go
into
the
ci
operator
into
the
step
registry
and
I
think
what
what
what
did
it
say
that
one
was
called
open
shift
e
to
e
and
then
test
okay
and
so
now,
we've
found
these
are
the
artifacts
that
go
with
that
test.
And
if
we
look
at
the
yaml
file
here,
this
will
tell
us
you
know
the
various
pieces
of
the
test.
Now
you
know
for
the
for
those,
you
know
who
kind
of
run
these
things
regularly.
A
This
might
start
to
look
like
we're
getting
into
the
kind
of
really
meaty
part
of
this,
where
it
talks
about
the
different,
the
different
pieces
that
need
to
be
run.
You
know,
there's
a
script
file
that
needs
to
be
run
here
and
then
there
are
several
pieces
of
tests
that
need
to
be
run.
So
this
is
how
you
would
start
to
unwind
what
is
actually
running,
and
where
can
I
find
this
code,
and
where
can
I
find
the
tests
that
are
running
now?
A
I
will
kind
of
shortcut
some
things
here
and
just
say
that
if
you're
looking
for
the
end-to-end
test
from
the
upstream,
those
are
mostly
contained
in
the
open
shift,
origin
repository
and
so
in
here
is
where
we'll
start
to
find
really
like.
You
know
the
the
heart
of
the
testing
tools,
and
so
when
you
see
those
things
that
says
big
network,
you
know
this
test,
and
maybe
maybe
this
is
a
good
case
for
us
to
go
back
and
look
here.
A
But
boxing
yeah
okay,
so
let's
let's
see
if
we
can
just
do
a
search,
I'm
going
to
close
a
couple
windows
here:
okay,
so
we're
in
the
we're
in
the
test
repository
here
in
the
origin
repository
and
let's
just
see,
let's
first
just
search
for
sig
network,
you
can
start
to
see
it's
going
to
tell
us.
Okay,
there's
some
pieces
here.
We
should
start
looking
at
so
you
can
start
to
understand.
This
is
how
we
would
go
back
through
and
start
to
understand.
A
Yeah,
so
this
is
something
that
we
run
into
on
a
day-to-day
basis
where
it's
like.
Well,
we
might
not
be
able
to
find
it
here.
So
at
this
point
I
might
need
to
check
the
repository
out
and
do
a
check
there.
Another
place
to
look,
though,
might
be
the
the
steps
registry.
I
don't
know
dennis.
Do
you
want
to
talk
about
the
steps
registry?
If
I
open
it
up
here.
A
B
A
A
B
For
some
reason
so,
but
yeah
here
should
be
also
a
graph
with
the
steps
yeah
here
which
comes
to
play
within
that
workflow.
A
progress
list
list
of
people
who
are
in
charge
for
that
particular
workflow
and
also
documentation
strings,
which
always
quite
useful
yeah.
Also,
these
websites
have
search
field
which
helps
a
lot
when
you
try
to
try
when
you
want
to
find
concrete
steps
or
workflows
so
try
to
type
ccm
install
here,
for
example,.
A
Okay,
now
this
was
something
that
confused
me
is
that
I
thought
you
could
just
look
for
any
test
here,
but
sometimes
the
tests
don't
show
up
and
I'm
not
quite
understanding.
Why
that
why?
That
is,
but
I'm
guessing
there's
a
difference
between
the
test,
I'm
looking
for
in
the
source
of
the
test
here.
B
A
Now
now
this
is,
I
find
this
interesting,
and
maybe
this
is
just
a
rendering
error
here
that
these
aren't
showing
up,
but
it
does
show
you
the
pre-steps
and
then
the
test
step
itself
and
then
the
post
steps.
Now,
interestingly,
I
just
searched
for
openshift
e
to
e
test
in
in
the
upper
left
upper
right
here
and
it
didn't
go
through,
but
it
is
here
and
we
can
find
it.
A
A
You
know
just
this
failure
that
we
had
here
this
website,
I'm
going
to
share
search.ci,
and
this
is
a
website
where
you
can
search
failures
that
are
happening
on
the
open
shift,
ci
infrastructure.
Now
this
is
not
specifically
looking
at
okd,
but
this
because
they
share
our
common
code
base.
This
is
a
place
you
could
come
to
start
searching
for
failures
to
see
if
things
have
happened
in
a
similar
way
to
other
things,
you've
seen
so
and-
and
you
can,
you
know
you
can
just
focus
on
a
specific
job.
A
A
So
this
high
number
alert,
slash,
fcd
high
number
of
failed
grpc
requests
should
not
be
at
or
above
we
can
start
to
look
and
see.
Okay
did
that
appear
in
other
places,
and
it
looks
like
maybe
this
you
know
we
can
see
it
here
coming
out
in
other
tests,
so
these
other
tests,
then
we
could
start
to
drill
into
those,
and
this
takes
us
back
to
our
prow
and
you
can
see
this
is
a
different
test
than
the
first
one
that
was
running.
A
This
is
some
openstack
test,
but
it
produced
a
similar
error,
and
so
we
might
look
here
to
see
if
there's
some
sort
of
similarity,
although
this
test
looks
like
it
failed
in
a
very
spectacular
fashion,
with
538
tests
failing
so
this
is
another
great
tool
that
you
can
use
to
kind
of
look
for
these
things
dennis.
Did
you
have
any
comments?
You
want
to
add
about
the
search
tool.
B
Yeah,
so
there
is
a
select
box
which
says
now
black
plus
issue
plus
junit,
so
there
is
also
search
around
build
logs
and
you
can
use
that
or
search
around
open,
open
shift
box
or
search
around
junit
results,
so
yeah
that
might
be
quite
useful,
but
sometimes
it
needs
some
place
around
around
keyboards
and
that
kind
of
stuff
so
and
time
to
time
it
produces
a
huge
output
and
yeah
that
takes
some
time
to
analyze.
The
output
of
these
two.
A
Yeah
I
was
gonna,
I
was
gonna
see
if
we
could
pull
like
a
log
line
here.
That
might
give
us
like
one
of
those
famous
huge.
You
know
outputs,
but
this
this
log
was
actually
very,
very
long.
Okay.
So,
let's
see
if
we
can
find
an
error
in
the
logs
here
and
I'll,
just
I'm
going
to
scan
very
quickly
looking
for
red
text,
okay,
so
we'll
look
here:
okay,
here's
a
failed
to
sync
config
map
cache.
A
A
We
could
come
here
and
put
that
here
and
we
could
use
the
regex
to
kind
of
build
this
out.
But
for
this
we
might
just
look
at
it
and
say:
okay,
so
here
we
say
we
said
found
and
I'll
increase
this
just
because
that
text
is
very
small
found
in
0.1
percent
of
runs
across
46
000,
total
runs
and
4
000
jobs.
So
in
this
way
this
could
become
a
very
verbose
tool
then,
because
now,
like
dennis,
had
just
suggested.
You
know
this
has
returned
us.
A
B
Yeah,
it's
also
always
useful
to
adjust
the
type
of
search
so
which
I
mentioned
before,
and
this
particular
one
was
found
in
a
bug.
I
guess
in
red
hat
bugzilla
yeah.
This
search
will
also
have
capabilities
to
search
across
box
issues
and
that
type
of
stuff,
so
just
for
us
to
not
open
duplicated
box
and
that
kind
of
things.
A
Yeah,
that
is
a
great
call
out,
dennis
and
and
it
when
it
says,
bug
here,
then
this
link
next
to
it
is
actually
going
to
take
us
to
the
bugzilla,
and
there
are
some
transitions
happening
with
bugzilla
right
now,
so
this
this
might
be
why
it's
taking
quite
a
while
to
load,
but
but
ideally
this
would
take
you
to
the
bug
that
is
linked
to
that.
So
again,
this
is
like,
like
you
said,
dennis.
A
So
at
this
point,
we've
kind
of
talked
a
lot
about
how
you
can
use
these
logs,
how
you
can
find
what's
happening.
This
is
really
like
an
introduction
and
we
could.
We
could
go
much
deeper
into
a
lot
of
these
topics,
but
at
this
point
I
would
like
to
stop
sharing
screen
and
I'll
invite
diane
to
come
back,
and
maybe
ask
us
if
there
are
any
questions
as
an
observer
is
watching
this
like
what
you
know
is
there
anything
specific
we
might
have
missed,
or
we
should
share.
C
Well,
I
think
one
of
the
this
is
great
by
the
way,
and
you
know
people
get
lost
in
these
log
files
and
these
tools.
So
it's
a
really
nice
step-by-step
approach
to
taking
and
diagnosing
some
of
these
failures.
But
what
would
if,
if
someone,
not
a
red
hatter
external
and
the
okd
working
group,
was
looking
at
these
and
had
diagnosed
something
or
found
something
of
interest,
what
would
the
best
way
for
them
to
reach
out
to
the
engineers
and
to
you
know,
log,
a
bug
or
log
an
issue?
C
A
So
I
will
log
back
in
once
more,
and
this
is
like
a
great
opportunity
for
anyone
who
wants
to
get
involved
in
the
community.
What
you're
going
to
want
to
do
is
is
create
either
a
red
hat
account
or
create
an
account
with
our
jira,
so
you
can
start
making
bugs
here
and
I'll.
I
have
to
be
a
little
careful
here,
because
I
want
to
make
sure
I'm
not
exposing
anything.
That's
too
sensitive
on
my
screen.
A
So
just
double
check
that
and
yeah
like
where
I
would
want
people
to
go
is
I
would
want
them
to
go
to
issues.redhat.com
and
create
an
account-
and
this
is
our
this
is
what
our
jira
looks
like
and
you're,
seeing
a
little
bit
of
the
inside
here.
So
hopefully,
I'm
not
exposing
any
too
sensitive
information,
but
what
you
would
want
to
do
is
click
on
the
create
button,
once
you've
got
your
account
made
and
you
want
to
find
open
shift
bugs-
and
this
is
where
you
can
put
a
generic
bug.
A
A
We
say
what
component
that
is,
and
this
is
cloud
compute
if
you
were
looking
at
just
the
machine
api,
but
you
would
see
a
sub
component
listed
if
there
was
some
other
piece.
So,
for
example,
if
we
look
at
the
auto
scaler,
which
is
another
piece
that
we
work
on,
you
will
see
that
in
the
owner's
file
we
have
listed
cloud
compute
and
the
sub
component
is
cluster,
auto
scalar.