►
Description
Deploying an application onto Red Hat OpenShift? The best-laid plans of mice and men don't always come to fruition, and as OpenShift administrators, we must be prepared to diagnose and fix deployment difficulties as they arise. Join us for this episode as we journey through the process of troubleshooting OpenShift when things go sideways. We'll cover some common problems, searching through logs, investigating unique pod issues, and more. So please join to learn more and ask us anything about troubleshooting Red Hat OpenShift.
About Ask an OpenShift Admin:
Do you have questions? We want to answer them! Every Wednesday at 11am ET, hosts Andrew Sullivan and Jonathan Rickard sit down to answer your questions about Red Hat OpenShift.
Follow us:
Andrew Sullivan , https://twitter.com/practicalandrew
Jonathan Rickard, https://twitter.com/jrocktx1
Subscribe to Red Hat's YouTube channel: https://www.youtube.com/redhat/?sub_confirmation=1
About OpenShift:
Red Hat OpenShift is an open source container application platform based on the Kubernetes container orchestrator for enterprise application development and deployment.
Openshift.com https://red.ht/33Fh6uV
OpenShift on github https://github.com/openshift/
OpenShift on redhat.com https://red.ht/2o4ccDk
#RedHat #OpenShift #ContainerPlatform
A
A
Yeah,
such
is
life
right,
say,
lavi,
so,
yes
stephanie,
it
has
been
a
lively
morning
for
a
number
of
different
reasons,
not
the
least
of
which
is
this
u.s
west
issue
happening
in
aws.
That
is
very
much
apparently
affecting
us
as
well.
You
know,
for
those
who
don't
know,
we
use
a
service
called
restream,
which
is
relies
on
some
aws
services.
Apparently,
so,
even
though
you
know
google
youtube,
you
know
it
doesn't.
Twitch
certainly
does
restream
apparently
does
so
apologies
for
the
technical
difficulties.
A
A
So
somebody
did
ask
about
you
know
using
geneve,
so
janie
ovn
kubernetes
does
use
geneve
for
the
tunneling
protocol
as
opposed
to
vxlan
with
openshift
sdn,
and
I
think
what
mark
was
getting
ready
to
say
before
he
got
cut
off
was
one
of
the
driving
factors
behind
the
change.
Is
the
community
behind
ovn
kubernetes?
A
You
know
openshift
sdn.
As
I
said,
it's
been
the
default.
It's
been
an
open
shift
since
like
3.0,
but
as
the
name
kind
of
implies,
it's
only
used
by
openshift,
so
the
community,
the
user
base,
is
as
substantial
as
openshift
itself
is,
which
is
not
to
say
that
it's
a
small
community,
but
when
we
compare
that
against
ovn
kubernetes
and
specifically,
you
know,
take
the
kubernetes
out
of
there.
Ovn
we're
talking
about
a
much
much
larger
community
of
folks.
So
just
within
the
red
hat
portfolio,
you've
got
things
like
open
stack.
A
A
You
know,
participate
in
that
we
can,
you
know,
join
and
ultimately
contribute
to,
and
you
know,
effectively
deliver
a
better
set
of
capabilities,
and
you
know
if
you
look
in
the
docs
there's
a
number
of
there's
a
comparison
chart
that
I
need
to
grab
a
link
to
that
lists
out
some
of
the
things
that
are
inside
of
there.
You
know
capabilities
available
with
ovn
that
aren't
available
with
openshift
sdn,
so
things
like
ipsec,
you
know
you
can't
do
the
node
to
node
encrypted
communication
with
openshift
sdn.
A
I
also
know
that
the,
and
so
the
performance
and
scale
team
does
a
lot
of
testing
with
the
two
and
does
a
lot
of
comparisons
with
them
and
the
you
know
performance-wise
scalability-wise.
As
far
as
I
know,
everything
I've
seen
is
that
they're,
basically
equal
rights.
They
they
trade
back
and
forth.
You
know
between
you
know
a
few
percentage
points
on
which
one
is
better
at
certain
things
than
others,
probably
the
biggest
difference
that
I've
seen
and
unfortunately
we
can't
share
like
the
graphs
or
anything
like
that.
A
We
can't
share
the
specifics
because
they
don't
let
us
for
whatever
reason,
but
the
biggest
one
that
I've
seen
is
it
takes
a
few
more
milliseconds
to
make
the
initial
connection
for
the
pod
to
the
sdn
right.
So,
instead
of
taking
like
10
milliseconds,
it
takes
12
milliseconds
or
you
know
whatever.
That
number
happens
to
be.
A
I
don't
remember
off
the
top
of
my
head,
but
yeah
performance,
throughput,
latency
wise,
you
know
they're
they're,
all
the
same,
or
at
least
at
least
the
same
I'll
say
if
not
better
with
ovn,
kubernetes
and
scale
is
certainly
an
a
positive
as
well
all
right
so
keeping
in
line
or
keeping
in
trend
with
being
an
interesting
and
lively
week.
So
last
week
it
was
the
aws,
the
u.s
east
outage
this
week,
monday
kicked
off
with
the
bang,
with
the
log
for
shell
vulnerability
being
announced.
A
A
A
You
know
we
know
about
them
for
days
or
weeks
beforehand.
They're,
just
you
know,
embargoed.
We
can't
talk
about
them.
You
know
they
don't
share
them
outside
of
a
very
small
group
of
of
engineers,
because
it's
a
security
vulnerability
and
then,
when
it
does
go
public
we
immediately.
You
know
as
soon
as
it
goes
public
the
cve
page
for
red
hat
has
here's
the
affected
products.
Here's
all
the
list
of
fixes
type
of
stuff.
This
one
was
not
like
that.
A
B
A
A
B
A
A
A
Yeah
so
again,
I'm
using
windows.
Today,
please
don't
judge
me.
Windows.
11
has
been
an
experience
so
far.
So
so
this,
oh
here's,
the
so
this
is
that
that
first
kcs
kcs
link
that
stephanie
had,
and
you
can
see
that
this
goes
through
and
it
walks
through
how
to
set
the
environment
variable
to
effectively
mitigate
the
vulnerability,
and
that
includes
doing
things
like
so
down
here.
We
set
the
operator
in
an
unmanaged
state.
A
So
that
way
it
will
not
basically
clobber
that
change
that
we
make
up
above
so
what's
happening
here
is
remember
with
most
things
openshift,
the
operator
manages
the
configuration
of
it.
So
if
I
go
in
and
I
customize
or
I
change
something
like
the
pod
environment
variables,
the
operator
will
reset
that
back
to
its
known,
good
configuration
so
until
and
I
believe
that
the
the
operator
fixes
have
been
released
by
now.
But
if
you
haven't
gotten
those
right,
if
you're
disconnected
or
something
like
that,
basically
you
can
go
in.
A
You
can
set
it
to
unmanage.
You
set
the
values
that
are
up
here
and
then
the
operator
won't
overwrite
those
changes
and
then
once
you
do
have
the
operator
update
in
place
kind
of
either
before
or
after
you
can
go
back
and
set
it
reset
it
to
managed,
and
then
the
operator
will
take
its
will
do
its
thing
and
apply
all
of
the
relevant
fixes.
A
So
that's
the
first
one
of
those
links.
That's
inside
of
here,
the
six
five
seven,
eight
four
two
one,
the
other
one
is
this
particular
security
advisory.
So
this
one
is-
and
this
is
the
5129,
so
this
is
for
you
could
see
openshift
logging
and
security
updates
for
openshift
logging
service
version
531.
A
A
A
Make
sure
that
you
update
this?
I
know
that
a
lot
of
us
you
know,
despite
it
being
the
holidays,
a
lot
of
us
are
going
to
be.
You
know,
working
on.
You
know
diligently
fixing
this
and
getting
everything
you
know
up
and
patched
and
making
sure
that
it's
it's
working
adequately
bmi
three.
I
see
the
ocp
update
graph
application
it
not.
It
was
not
working
yesterday
because
of
update
service
issues.
That's
interesting!
I
wonder
why
I
hadn't
heard
that
there
was
an
outage
so
hopefully
well.
A
B
A
B
A
Talked
about
this
last
time
with
4.7
to
4.8,
you
know,
usually
the
the
window
for
going
from
for
stable
updates
between
minor
versions,
so
4.8
to
4.9,
4.7,
4.8
4.6
to
4.7
is
somewhere
between
50
and
60
days.
I
think
last
week,
when
we
talked
about
this,
we
were
right
at
50
days
or
51
days,
something
like
that.
So
keeping
in
line
with
historical
trends.
We
now
have
those
updates
available
and
the
the
telemetry
that
they
have
shared
with
us.
It
looks
pretty
good,
you
know
we
we
haven't.
A
I
don't
think
we've
seen
quite
the
same,
not
number
but
magnitude
of
issues
that
we
had
seen
before
and
even
that's
not
really
fair
right
before
it
was.
We
were
encountering
specific
issues
on
specific
platforms
like
the
vmx
net
3
driver
with
vsphere
that
type
of
stuff
amit.
Can
we
make
efs
provisioner
connect
to
multiple
efs
servers
at
once
and
provision
apv.
Accordingly,
we
are
on
3.11
and
it
is
tech
preview.
A
So
I
will
admit
I
am
not
familiar
with
efs,
so
my
understanding
is
that
amazon
ships
an
efs,
csi
provisioner
that
will
work
with
openshift
4.,
whether
or
not
you
can
connect
that
to
more
than
one
is,
is
not
known
to
me,
and
it
may
be
a
question
for
amazon
themselves
and
3.11
it
I'm
definitely
unfamiliar
with
how
it
works
on
that
platform.
Johnny.
B
A
Okay,
so
amit
you
can
reach
out
to
me
so
twitter,
practical,
andrew
or
email
andrew.sullivan
redhat.com.
If
you
want
to
reach
out
to
me
with
your
question,
we'll
see
if
we
can
track
down
some
better
information
on
that
get
something
more
authoritative
than
andrew
thinks
or
johnny
thinks
and
and
we'll
let
you
know
with
that,
also
keep
an
eye
out
for
the
blog
posts.
A
So
I
I
am
certainly
not
going
to
hijack
their
pto
just
to
get
a
blog
from
little
me
published,
but
keep
an
eye
out
in
the
new
year,
we'll
have
a
whole
bunch
of
those
kind
of
dump
and
then
you'll
be
able
to
go
back
and
search
and
find
all
of
that
so
yeah.
So
please,
please
don't
hesitate
to
ask
your
questions.
We'll
do
our
best
to
get
those
answered.
A
A
I
saw
it
do
this,
you
know
we
we've
all
had
those
experiences
where
it's
you
know
what.
Why
do
you
stand
on
your
left
foot
and
hop
around
in
a
circle
three
times
before
you
do
this
well
because
last
time
I
did
that
it
worked
and
it
fixed
the
problem.
You
know,
or
the
infamous
I'm
gonna
poke
fun
at
windows
today,
because
I'm
using
windows
right,
the
infamous
reboot
reboot,
fixes
everything
right.
A
It
is
a
science
in
addition
to
an
art,
and
what
I
mean
by
that
is,
there
is
a
whole
set
of
kind
of
philosophies
and
things
that
we
can
apply
to
help
make
that
troubleshooting
process
a
little
more
cohesive,
a
little
more
organized
and
a
little
more
planned,
even
though
often
times
when
we're
doing
it.
It
is
a
very
chaotic
and
hectic
thing
right.
Nobody
likes
to
get
called
in
in
the
middle
of
the
night
because
something's
wrong,
and
then
you
know
you're
trying
to
figure
out.
A
A
So
the
other
thing
that
I
will
quickly
point
out
is
yes:
for,
like
two
or
three
weeks
now
we
were
talking
about
having
a
vmware
guest
on
today.
Unfortunately,
they
had
a
last-minute
conflict,
so
they
were
unable
to
join,
but
we
will
definitely
have
them
join
us
in
the
new
year.
We've
we've
already
rescheduled
that
for
the
middle
of
january.
A
So
please,
if
you,
if
you're
interested
in
that,
if
you're
interested
in
any
of
the
content
that
we
have
be
sure
to
subscribe
on
whatever
platform
you're
watching
us
on,
so
you
can
get
those
those
updates
all
right,
troubleshooting,
so
johnny
you
and
I
we
were
both
administrators
for
a
long
time.
You
know
you
were
on
the
consulting
side.
I
know
that
you
did.
A
B
Yeah
for
sure
so
like
when
you
say
it's
an
art
and
a
science.
It
definitely
is
right
because,
like
over
time
as
you
establish,
you
know,
like
your
maturity
with
how
you
you
do
things
right,
you
come
up
with
a
process
that
works
for
you,
because
it's
like
okay,
well,
I
know,
like
you
said
I
started
here.
Let
me
start
here
so
for
me.
What
what
I
typically
do
is
when
I
learn
you
know,
especially
like
openshift4,
with
all
the
different
components.
B
I
start
like
the
very
bottom,
all
right
where,
where
do
I
start
at
the
install
config?
Okay,
if
you
watch
me,
do
my
disconnected
open
shift
install
then
you
realize
that,
like
I
was
blowing
up
on
the
install
config
and
it
was
because
I
had
spaces
in
my
pull
secret,
but
like
so
I
I
just
started
the
basics
right.
I
go
this
and
I
keep
it
as
simple
as
possible,
like
okay,
here's,
the
problem
that
I'm
having.
B
Let
me
work
one
step
backwards
and
then,
if
I
can
get
beyond
that,
then
I
try
and
work
one
step
forwards
and
then
I
just
try
and
identify
where
the
issue
is
and
then
once
I
identify
the
issue,
then
obviously
I
implement
the
fix
and
then
I
try
to
understand
why
I
had
that
issue
and
then
what
happens
is
when
you
understand
why,
then
it
helps
build
that
logic
into
troubleshooting.
So
the
next
time
you
see
it
you're
like
okay,
I'm
not
getting
to
this
point.
So
most
likely
it's
because
of
this.
B
A
I
have
to
say
that
one
of
the
most
terrifying
things
for
me
was
always
well.
It
just
started
working
again.
Well.
Why?
Well,
I
don't
know
yes,
but
then
we
don't
know
what
broke
it
and
we
want
to
figure
out
what
caused
the
outage
or
what
caused
it
to
break
so
that
we
can
stop
it
from
happening
in
the
future.
A
A
B
It's
the
blue
coupe
service
on
the
bootstrap
node.
That
to
me
gives
you
so
much
information,
because
even
as
it's
going
through
and
it's
if
it
fails
at
some
point,
you'll
get
a
lot
of
info
out
of
there.
The
next
place
I
would
like
during
the
bootstrap
process
is
the
kubelet
service,
because
it's
you're
going
to
get
a
lot
of
information
out
of
there
as
well,
but
you'll
see
any
errors
when
it's
communicating
with
aws
or
azure
or
whatever
from
from
your
provider.
Standpoint
you'll
start
seeing
a
lot
of
information
out
of
there.
B
Then,
if
I'm
really
have.
If
it's,
if
it's
a
deeper
issue,
then
I
would
look
at
chris
ctl.
So
chris
ctl
runs
a
lot
like
pod
man
or
even
like
a
docker.
You
know.
So
if
you
do
like
cry,
ctl,
ps
and
cry
ctl
ps
once
a
and
cry
ctl
logs,
and
so
that
way
you
can
actually
follow
the
logs
on
the
containers
themselves.
B
So
that
way,
you
can
actually
isolate
down
to
the
thing
that
you
think
you're
having
a
problem
with
like
the
bootstrap
api
server,
for
example,
and
then
you
can
look
in
there
and
see
where
things
are
going.
A
good
example
where
you
could
do
that
is
if
you're
doing
a
single
node
open
shift,
install
and
you're.
B
Like
I
really
don't
know
what's
going
on,
you
can
actually
go
look
at
the
cry:
ctl
logs
for
the
cluster
version
operator
and
then
you'll
get
a
ton
of
information
on
where
the
cluster
is
deployed
from
or
where
it's
at
in
this
deployment
process,
and
if
it's
having
issues
like
it,
can't
resolve
something
or
whatever.
So
then,
moving
on
to
like
the
day-to-day
cluster
operations,
we
have
a
lot
of
customers
that.
A
So
I'm
gonna,
I'm
gonna,
interrupt
you
real
quick,
so
I
I
think
we
still
might
be
having
some
random
issues,
so
I'm
gonna
post
a
link.
I
posted
it
on
youtube.
I'm
also
posting
it
into
twitch,
because
I
didn't
see
it
pop
up
on
our
restream,
so
okay,
but
so
that's
the
overall
troubleshooting
article
for
openshift.
So
this
is
maintained
by
support.
It
has
a
link
to
a
bunch
of
other
articles.
We've
talked
about
it
before
here
on
the
stream
of
just
hey.
This
is
a
good
starting
point
and
good
resource.
A
A
A
So
you
know
right
now,
if
I'm
trying
to
deploy
4.9,
I
would
come
in
here
and
I
would
look
at
the
4.9
branch
to
determine
what
that
precise.
Behavior
is
yeah.
There's
a
bunch
of
different
and
you
know,
comes
from.
Experience
comes
from
learning
and
all
of
that
there's
a
bunch
of
different
components
that
go
into
the
install
process.
A
You
know
the
installer
is
one
component
machine
config
operator
is
another
component
which,
if
we
just
go
back
to
the
we
do
machine
gig
we've
got
machine.
Config
operator
machine
api
operator
is
another
one.
You
know
they
all
play
a
role,
so
just
it's
one
of
those
like.
Yes,
it
takes
time
right.
We
all
have
to
learn
how
to
do
these
things.
I
also
want
to
make
sure
to
highlight
and
to
remind
folks
that
you
can
always
open
a
support
case.
I
think
that's
that
that
is
always
a
very
good
option.
A
When
you
are
encountering
issues
right,
don't
don't
draw
out
an
outage
longer
than
needed
or
or
you
know
have
issues
longer
than
needed
when
you
can
open
a
support
case
and
and
get
help
or
you
can,
you
know,
reach
out
to
the
various
community
stuff
so
davo?
What
is
your
most
loved
feature
in
shifts?
That's
a
good
one!
I
don't
know
I'll
say
that
I
always
because
I'm
near
and
dear
to
the
installer,
I
do
a
lot
with
the
installer
I
find
deploying
openshift
once
you
have.
The
prerequisites
in
place
super
easy,
especially
ipi.
B
A
Yeah
it's
I
I
do
periodically,
because
I
I
work
with
the
community
and
do
a
lot
of
other
things.
I
do
periodically
go
in
and
deploy
kubernetes.
You
know
just
vanilla
kubernetes
with
you
know,
cube
adm
or
something
like
that
and
as
complex
as
openshift
is
some
things
when
they
work
right,
which
is
the
vast
majority
of
the
time.
You
know
it
is
really
easy.
Yeah.
So.
A
Yeah
yeah,
I
also
you
know
if,
if
you
want
to
call
it
that
the
community
around
openshift,
you
know
the
kubernetes
slack
channel,
there's
openshift,
you
know
channel
or
kubernetes
slack
team,
there's
openshift
channels
that
a
bunch
of
folks,
you
know
hang
out
in-
and
you
know
always
willing
to
answer,
questions
and
stuff
like
that.
You
know
all
of
the
github
issues
and
stuff
like
that.
You
know
I,
even
as
an
employee,
I'll
open
up
issues
on
github,
and
you
know
we
get
responses
very
quickly
from
the
engineers
so
yeah.
A
That's
awesome,
all
right,
so
just
to
recap:
real
quick.
So
with
the
install
process
connecting
to
the
bootstrap
nodes,
I
guess
it
depends
on
where,
in
the
process
you
are
connect
to
the
bootstrap
node
check
the
bootcube
log.
If
you
sshn
there'll,
be
a
very
helpful
and
part
of
the
mesh
message
of
the
day
is
the
exact
journal
control
command?
You
can
copy
and
paste
that
in
yeah
beyond
that
check
the
check
the
the
relevant
operator
logs.
A
You
know
if
it
doesn't
make
it
that
far.
If
you're
having
issues
you
know
instantiating,
you
know
just
getting
the
control
plane
running
check
the
pods
using
things
like
creek
cry:
cuddle,
creek,
cuddle,
cree
ctl,
whatever
I'm
gonna,
I'm
gonna
offend
everyone.
As
I
go
through
that
right,
however,
you
think
it's
pronounced,
I'm
gonna
butcher
it
so
yeah,
so
yeah
check
so
c-r-I-c-t-l
cry.
Cuddle
is
used
very
similar
to
how
you
would
pod
man
or
or
docker
to
do
like
a
cry.
B
A
B
Yeah
definitely
just
looking
at
the
the
operator
itself.
You
know
so
going
into
the
namespace
trying
to
figure
out
like
what
what
the
status
of
the
pods
are
so
doing
like
an
oc
get
pods,
but
one
thing
that
I
think
a
lot
of
people
overlook
is
oc
describe,
and
you
made
a
note
of
this
in
in
the
document
which
I
was
like
yeah.
This
is,
you
know,
plus
100
on
that,
because
oc
describe
pod
and
then
the
pod
name
will
give
you
so
much
information
about
the
deployment
itself.
B
A
Yeah
and
this
the
not
scheduled
one
is
always
the
most
baffling
and
the
most
helpful
right,
because
it's
you
know
oh
unable
to
schedule,
you
know
zero
of
12
nodes
available.
You
know
three
have
taints,
you
know
six
have
not
enough
capacity.
You
know
three
it'll
tell
you
like
exactly
what
the
issue
is
exactly
in
many
of
those
cases
or
at
least
give
you
a
good,
strong
pointer
towards
why
it
can't
schedule
that.
B
Exactly
and
in
case
you
don't
know,
typically
scheduling
is
either
selectors
labels,
taints,
tolerations
or
resources.
You
know-
and
I
think
that's
probably
the
biggest
one
that
we
see
with
logging
is
that
you
know
people
try
and
provision
small
and
then
put
a
big
service
like
cluster
logging
on
top
of
it,
and
then
they
don't
tune
the
instance
down
enough
to
where
it
can
actually
fit
on
the
nodes,
and
so,
for
example,
like
elastic,
I
think,
needs
like
six
gigs
of
ram
out
of
the
box
right.
A
Yeah,
no
in
a
perfect
example,
you
know
was
it
last
week
or
two
weeks
ago
we
were
showing
single
node
open
shift
and
I
deployed
a
single
node
openshift
instance
with
six
cpus,
and
I
wasn't
I
was
doing
other
things
I
keep
banging
into
the
microphone
arm,
so
I
was
doing
some
other
things
and
I
came
back
and
I
wanted
to
deploy
acm
onto
my
single
mode,
openshift
instance.
So
I
deploy
everything
and
I've
got
a
bunch
of
pods
that
are
just
in
the
pending
state.
A
What's
going
on
here,
like
you
know,
forcefully
terminate,
you
know,
restart,
what's
good
and
I
finally
oc
described,
I
didn't
have
enough
cpu
yeah
so
and
it
wasn't
and
I'm
it's
one
of
those
like
it
was
only
using.
I
think,
like
actual
utilization,
was
two
cpus,
but
the
requests
so
we're
over
that
yeah.
B
Yep
and
and
dmi
three
and
miss
I'm
sorry,
please
don't
don't
hate
me
for
that
yeah!
Oh
see,
you
got
events
as
another
one,
that's
it!
That's
the
other
one
that
we're
like
alluding
to
as
well
right.
That'll
that'll,
give
you
a
lot
of
information,
but
yeah.
Those
are
good.
Events
in
oc
describe
are
outstanding.
A
Yeah,
I
I
don't
think
so.
Yes,
you
can
filter
it,
but
it
requires
using
effectively
a
json
path.
Andrew,
as
christian
can
attest
to
andrew,
is
terrible
at
json
and
json
path,
queries,
but
you
can
use
that
in
the
output
so
as
a
part
of
the
command
to
filter
out
what's
there,
and
you
can
do
that
directly
through
the
oc
command,
you
don't
have
to
pipe
it
into
jq
or
anything
like
that,
although
that
is
an
option.
B
A
A
A
You
can,
or
you
should
also
check
operator
lifecycle
manager.
If
the
operator
itself
is
having
issues
so
maybe
olm
is
unable
to
update
it.
For
some
reason
maybe
olm
is
unable
to
deploy,
because
an
operator
is
just
pods
right,
just
like
anything
else
in
kubernetes,
so
maybe
olm
is
unable
to
fully
instantiate
the
operator
and
then
last
but
not
least,
and
definitely
the
least
common,
unless
we're
talking
about
updates
or
upgrades
is
cluster
version
operator.
A
So
if
it's
one
of
those
core
operators
like
if
you
do
an
oc,
git
co
oc
get
cluster
operator,
those
are
all
managed
by
cluster
version
operator.
So
you
would
want
to
check
in
you
know
cluster
version
operators
logs
stuff,
like
that,
if
it's
one
of
the
core
ones
I
quin
qui,
I'm
gonna
just
go
with
quinn
for
that
username.
My
apologies!
When
you
have
a
stateful
workload
on
your
cluster,
how
would
you
upgrade
slash
evict
the
underlying
node
with
minimum
impact
towards
the
workload?
A
So
there
is
no
minimum
impact
is
essentially
the
pod
is
terminates
and
then
is
rescheduled
and
reinstantiated
on
a
different
node.
That's
kind
of
the
way
it's
always
going
to
be
so
minimum
impact
would
be
achieved
through
something
like
a
redundant
service.
So
there's
you
know
two
or
three
or
five
or
however
many
instances,
so
that
there
is
no
one
single
point
of
failure
and
then
so
assuming
that
that
is
true.
A
So
when
we
do
an
upgrade
and
it
has
to
go
through
and
it
does
a
a
cordon
and
drain
of
each
node
in
the
cluster,
you
know
it
knows
I
can't
affect
so
many
nodes
or
I
can't
affect
both
nodes
at
the
same
time
that
have
this
pod
running
or
this.
This
application
running
yeah
type
of
thing
so
pod
disruption
budgets
can
be
a
huge
benefit.
A
B
Yeah,
it's
always
look
at
generally,
it's
going
to
be
an
image
pull
back
off.
So
that's
going
to
check
your
registry
check
your
image,
check
your
tag
and
then
or
it's
going
to
be
scheduling
right.
So
that's
going
to
be
team.
Tolerations
labels,
node
selectors,
all
that
stuff.
One
last
thing
that
we
we
kind
of.
I
was
trying
to
get
to
it
a
second
ago,
but
it's
it's.
B
If
you
turn
your
clusters
off
and
you
turn
it
back
on
and
say
over
a
weekend
check
your
csrs,
so
oc,
adm
or
oc
get
csr.
Because
what
happens
is
they
will
expire
over
like
a
certain
amount
of
time?
I
can't
remember
what
the
timeline
is,
but
a
lot
of
times
your
notes,
your
cluster
will
come
back
up
and
it
seems
like
it's
fine,
but
your
nodes
are
not
ready.
A
Yeah,
I
always
try
and
remind
folks
to
look
at
your
application.
Logs
too,
I've
worked
with
a
number
of
of
customers
who
it's
you
know
they're
like
way
down
in
the
weeds,
troubleshooting
the
sdn,
and
why
isn't
the
sdn
allowing
this
traffic
request
and
it
turns
out
it
was
like
if
you
look
in
the
application
log,
it
just
has
the
wrong
service
name
for
what
it's
trying
to
talk
to
or.
B
Yeah,
absolutely,
and
especially
as
we
get
like
more
advanced
right,
like
it's
just
harder
to
keep
it's
harder
to
think
that
small,
and
then
you
know
that
you
know
you
know
like
as
we
get
further
further
along
right.
Okay,
it's
got
to
be
this,
but
it's
it's
just
start
at
the
base
and
then
work
your
way
up
come
up
with
the
process
where
you
start
at
the
bottom.
So
that
way
you
actually
go
through
it.
Every
time.
A
A
You
know,
I
I
never
when
I
started
at
red
hats.
You
know
when
I,
when
I
started
this
career
in
tech
marketing
with
eight
nine
years
ago
now
I
I
never
thought
I
would
be
sitting
here.
You
know
talking
with
an
audience
of
folks
on
a
weekly
basis
being
able
to
interact
with
being
able
to
to
do
all
of
these
amazing
things
with
all
of
you,
amazing
people.
A
So
thank
you
so
much
have
a
great
holiday
season,
regardless
of
what
you're
doing
you
know
or
what
you
celebrate
or
don't
celebrate,
or
anything
like
that.
I
hope
that
the
end
of
the
year
is
really
phenomenal.
For
you
please
stay
safe,
you
know,
covet
is
still
covered
and
still
doing
its
thing,
and
you
know
ultimately
again.
Thank
you
so
much
to
everybody.
Johnny.
Thank
you
for
joining
me
this
year.