►
Description
OpenShift 4.12 added MachineSet functionality for control plane nodes on some infrastructure types. But, what does that mean? What benefit do they provide? Today we'll take a look and explore this new feature to see how it makes things like recovery from failure easier than ever before.
Infra nodes, which host core services for OpenShift, are not configured "out of the box", but provide an important - and valuable - capability: isolating those services on nodes that don't require entitlements. We'll demo how to configure infra nodes and how to assign workload to them.
Want to learn more?
Try Red Hat OpenShift: https://red.ht/3uVxkya
Follow us:
Andrew Sullivan https://twitter.com/practicalandrew
Jonathan Rickard https://twitter.com/jrocktx1
OpenShift https://twitter.com/openshift
Subscribe to Red Hat's YouTube channel: https://www.youtube.com/redhat/?sub_c...
About Ask an OpenShift Admin:
Do you have questions? We want to answer them! Every Wednesday at 11am ET, hosts Andrew Sullivan and Jonathan Rickard sit down to answer your questions about Red Hat OpenShift.
About OpenShift:
Red Hat OpenShift is an open source container application platform based on the Kubernetes container orchestrator for enterprise application development and deployment.
OpenShift on redhat.com https://red.ht/3rYz5ZS
OpenShift on GitHub https://github.com/openshift/
#RedHat #OpenShift #Kubernetes #ContainerPlatform
A
B
Thank
you,
hello,
and
welcome
to
this
week's
ask
an
openshift
admin
office
hours
live
stream
and
you
think
97
episodes
in
I
could
get
the
name
right
on
the
first
trial
without
having
to
engage
my
brain,
never
yeah,
hello.
Everyone
welcome,
my
name
is
Andrew
Sullivan
I
am
co-host
along
with
and
joined
by,
my
very
good
friend
Johnny
Rickard.
How
are
you,
sir
I'm
good
buddy?
How
are
you
it's
another
beautiful
Wednesday?
B
B
That
we
want
to
look
at
today
and
see
a
demo
of,
and
then
the
other
one
is
infra
nodes.
Infra
nodes
are
something
that
seem
to
come
up
pretty
often
a
lot
of
different
questions
for
me
like
when
I
see
questions
about
infra
nodes,
it's
about
50
50
between
how
do
I
configure
how
do
I
use
them
for
nodes
and
what
am
I
allowed
to
run
on
infra
nodes,
so
we've
Johnny
and
I.
B
We
we
talked
about
this
a
little
bit
yesterday
of
like
maybe
we
should
start
doing
some
for
lack
of
a
better
term
back
to
basic
stuff.
It
occurred
to
me
that
we
haven't
done
like
an
install
I
mean
we've
done.
We've
done
some
install
things
right.
We
talked
about
like
a
youtube-based
installer,
we've
talked
about
assisted
installer.
You
know,
we've
shown
IPI
and
a
couple
of
different
things,
but
we
haven't
done
like
Back
to
Basics,
vsphere
UPI
or
you
know
bare
metal
type
of
install
I.
B
It
might
have
been
the
first
one
of
the
first
10
episodes
of
The
Stream.
You
know,
and
that
was
gosh
three
years
ago
coming
up
on
three
years
ago,
so
it
it
feels
like
it's
time
to
go
back
and
start
looking
at
some
of
that
stuff.
So
for
anybody
who's
out
there
who's
watching
the
stream
either
live
or
recorded.
B
A
A
That's
my
good
friend
Lester
likes
to
say
you
know
it's,
it's
good
to
have
like
a
solid
base
of
Foundations
right
or
solid
foundation
of
fundamentals.
You
know
so
you,
like,
okay,
here's,
how
I
created
a
basic
application
to
make
sure
that
my
cluster
is
doing
what
it's
supposed
to
be
doing.
Here's,
how
I
know
that
my
cluster
is
up.
You
know
just
yeah,
have
that
nice
fundamental
bass
line
and
then
move
on
from
there.
B
B
And
you
know,
of
course
that's
going
to
be
different
for
everybody,
everybody
out
there.
You
know
what
your
requirements
are.
What
all
that
type
of
stuff
is.
You
know,
for
example,
some
folks
I'm
going
to
pick
on
the
logging.
You
know
some
folks,
logging
is
absolutely
critical
must
be
there.
You
know
cannot
cannot
have
a
gap
has
to
be
bulletproof
type
of
thing
because
you
know
Regulatory
Compliance
for
other
folks.
It's
logging
would
be
great,
but
it's
you
know
a
lesson.
You
know
it's
not
a,
not
a
tier,
a
cluster
right.
B
You
know
the
stuff
that's
happening
here.
It
can
go
away,
you
know,
or
the
applications
have
their
own
log
mechanisms.
We
don't
need
to
do
it
at
the
cluster
level.
So
but
anyways,
that's
a
it's
an
interesting
concept.
Right,
login
Fields
will
kill
you
that
doesn't
sound
good
anyways.
So
this
is
one
of
the
office
hours
series
of
live
streams
here
on
red
hat
live
streaming,
which
means
that
Johnny
and
I
are
here
for
you
all.
So
anything!
That's
on
your
mind.
Anything
you
want
to
ask
about
anything.
B
You
want
to
talk
about,
feel
free
to
bring
it
up
in
chat,
whatever
platform
you're
on
whether
whatever
YouTube
channel
twitch
Etc,
we
it'll,
get
rebroadcast
across
all
of
them
so
feel
free
to
ask
and
we'll
do
our
best
to
answer
those
questions.
That
being
said,
we
of
course
always
have
a
topic
that
we
talk
about
and
fill
in
if
you
will
with,
but
we're
happy
to
talk
about
those
questions
as
well,
and
that's
really
why
we're
here,
if
you're
not
watching
us,
live,
please
still
reach
out
to
us.
B
We'd
love
to
hear
from
you
all.
We
love
to
get
those
questions
a
lot
of
times,
we'll
bring
them
up
here
on
the
stream
to
to
chat
about
that
and
kind
of
make
sure
that
it
gets
out
there.
You
can
always
reach
out
to
me
via
email
at
andrew.sullivan
redhat.com,
or
you
can
reach
out
to
me
on
the
various
social
media
platforms
at
practicalandrew,
so
Twitter
and
Reddit,
and
all
of
the
others.
I
should
really
join.
Mastodon
I
feel
like
that.
B
I
still
have
this
sticky
note
on
the
monitor
of
all
the
different
places,
renetti
slack
and
like
all
these
other
things
of
like
where,
where
we
are
present
at
there's
one
here
that
I
can't
actually
read,
because
my
oh
LinkedIn,
because
my
handwriting
is
terrible,
you
know
I,
don't
ever
write
things,
I
type
things
so
yeah,
please
don't
hesitate
to
reach
out
love
to
get
your
emails
love
to
hear
from
you.
You
can
also
reach
out
to
Johnny.
B
B
A
I
think
that
there's
something
in
I
feel
like
there's
something
in
the
works
within
get
Ops
itself,
where
they're
going
to
make
machine
set
management
a
lot
easier
like
they're,
just
going
to
simplify
the
whole
process
so
like
this
is
a
good
stop.
Gap
I!
Think
for
now,
and
I
mean
you
know
up
until
now
or
until
they've
released
that
future.
But
I
know
I,
remember,
hearing
something
about
machine
set
management
being
simpler
with
get
Ops.
B
A
B
B
Let's
see
so
the
first
thing
I
wanted
to
bring
up
here
is
actually
one
I
didn't
put
in
the
list
Johnny,
but
I'm
gonna
quickly
bring
it
up,
which
is
upgrades
in
the
stable
channel
from
4.11
to
4.12.,
so
you'll
notice
that
they
are
not
there.
Yet.
Let
me
turn
this
guy
on
and
go
to
upgrade
no
you're
not
gonna
fill
in.
For
me,
fine.
B
B
B
B
If
I
go
into
the
Cincinnati
graph
data
Repository
and
looked
at
blocked,
edges,
I
can
come
down
here
and
see
come
on
while
it's
scrolling
so
much
scrolling,
so
I
want
4.12.7.
So
we
see
that
there's
two
blocked
Edge
identifiers
in
here
for
4.12.7,
so
I'm
going
to
pick
one
of
these
and
it's
saying
from
everything
4.11
to
4.12.7,
it's
blocking
that
edge
and
it's
blocking
it.
For
this
reason,
and
then
I
don't
know-
we've
talked
about
this
before
Johnny
of
in
in
modern
openshifts.
B
It
uses
promql
inside
of
the
cluster
to
identify
whether
or
not
your
cluster
is
affected
by
these
things.
So
I
always
thought
that
was
really
interesting
because
even
you
know
like,
even
if
you
have
a
connected
cluster,
we
don't
get
the
information
that
we
need
to
be
able
to
say
yes
or
no.
Your
cluster
is
affected
by
this.
It
has
to
be
done
inside
the
cluster
yeah.
B
So
what's
interesting
about
this
one-
and
you
can
kind
of
read
through
the
notes
here
if
your
cluster
was
deployed
as
an
open
shift,
4.1
cluster
so
way
back.
When
do
you
remember
openshift
4.1
Johnny?
It
was
I.
Don't
I,
don't
think
I
know
AWS
was
supported,
I
think
vsphere
was
supported,
I,
don't
know
what
other
like
there
wasn't
a
lot
of
platforms.
A
B
For
me,
until
like
four
three
four
four
one
installing
so
you
got
AWS
bare
metal
or
vsphere
so
yeah
it
was.
It
was
way
back
when
so.
If
your
cluster
was
was
deployed,
4.1
and
has
been
upgraded
over
the
years
and
now
reaches
you
know:
4.11
4.12
status
there's
a
chance
that
it
doesn't
have
this
particular
flag
on
podman
I
think
yeah
podman.
B
So
this
is
one
of
those
times
where
you
know
the
conditional
updates.
That's
the
word
I
was
looking
for
comes
into
play
of
as
the
administrator
we
can
go
in
and
we
can
look
and
say:
oh
my
cluster
was
deployed
four
dot,
11
or
4.10
or
whatever.
You
know
this
clearly
doesn't
affect
me.
So
yes,
please
go
ahead
and-
and
you
know
damn
the
torpedoes
Admiral
Farragut
by
the
way
so
anyways.
If
we
come
back
here
and
now,
I
lost
my
tab
or
I
think
I
just
closed
my
tab.
B
This
is
one
way
that
we
can
see
all
the
different
stuff
that
happens
inside
of
here
and
get
an
idea
of
things
that
might
be
blocking.
I
do
know
that
the
team
you
know-
normally
we
say
45
to
60
days
after
GA-
is
when
stable
upgrades
between
minor
versions
are
unlocked.
I
think
we're
a
little
bit
past
that
60-day
Mark
at
this
point,
but
I,
know
they're
working
diligently
to
get
past
that
the
45
to
60
is
not
a
guarantee.
B
But
yeah
here
you
can
see
here's
the
issue
leaked
machine,
config,
I,
don't
know
that's
an
interesting
one.
It
could
be
that
no,
if
it's
blocked
from
earlier
versions
of
4.11.
Sometimes
what
will
happen
is
the
like,
we'll
end
up
in
a
scenario
where
4.10
dot
I'm
going
to
use
a
random
number
46
is
blocked
from
upgrading
to
4.11
dot,
something
even
though
4.10.485
isn't,
and
that
happens
because
the
latest
Z
stream
for
in
this
instance
4.10.
The
core
OS
version
is
actually
ahead
of
the
latest
z-stream
version
for
4.11..
B
B
B
I
spent
the
first
six
months
that
I
was
at
Red
Hat,
just
trying
to
wrap
my
head
around
openshift
4.,
and
so
you
know,
upgrading
even
at
4.2
is
pretty
impressive.
There
I
will
I
will
kick
you
because
it
was
such
a
paradigm
shift
with
operators
and
the
way
the
deployment
is
done.
You
know
just
the
install
and
the
way
cluster
management
is
done
with
you
know.
All
of
these
other
things
was
such
a
dramatic
shift.
B
A
Too
me
too,
especially
like
ansible
I.
Have
an
inventory
I've
got
it
perfected.
It
works
100
of
the
time
it's
worth
99.99
of
the
time.
Yeah
that
part
to
me
was
like
it
made
sense.
It
took
me
a
while
to
like
once
I
figured
out
like
okay,
here's
the
step.
You
know,
here's,
you
know
here's
my
blog
diagram
of
how
this
actually
works
and
it
helped
me
debug
it.
You
know
to
be
honest
and
once
I
figured
that
stuff
out
I
was
like.
Oh,
this
is
actually
not
so
bad
yeah.
B
Yeah,
it's
I
spent
a
lot
of
time.
Writing
my
own,
like
cheat
sheet
on
operators
and
stuff
yep.
Okay,
I
spent
longer
on
that
than
I
intended
to
so.
Let's
see
the
next
one
that
we
wanted
to
talk
about.
This
is
one
that
you
highlighted
to
me.
Like
I
saw
some
some
noise
about
it,
but
I
hadn't
actually
gone
and
read
this
thing
until
you
pointed
it
out,
I,
don't
know
if
you
want
to
highlight
anything
here,
yeah.
A
It
really
kind
of
like
it
lines
up
with
you
know
what
we
were
talking
about
last
week
with
you
know,
trying
to
restore
STD
from
a
backup,
but
like
Reddit
on
the
14th
I
think
we
even
mentioned
this
vaguely,
or
a
little
bit
early
last
week
too,
about
Reddit
being
down
at
some
point:
well,
their
kubernetes
cluster.
They
want
to
upgrade
it
and
during
the
upgrade
they
had
a
catastrophic
failure
essentially
and
they
they
posted
their
entire
postmortem
online
as
a
subreddit,
and
it
is
amazing,
like
it's,
it's
very
detailed.
A
It's
it's
a
good
read,
you
know,
there's
a
lot
of
humor
in
it
and
the
the
gist
of
it
is
that
one
of
the
core
problems
is
when
the
the
labels
changed
from,
say,
master
to
control,
plane,
they
didn't
catch
it,
and
so
the
the
machines
we're
trying
to
deploy
they're
trying
to
deploy
into
certain
things.
They
thought
it
was
Calico
at
first,
but
they
figured
out
that
Calico
was
working
and
then
when
they
they
saw
the
logs.
A
Like
oh
snap,
it's
trying
to
hit
a
label
that
doesn't
exist,
but
they'd
already
restored
at
that
point.
So
it
just
goes
to
show
that
you
know
like
it's.
Upgrades
are
difficult
in
on
any
platform,
and
you
know
trying
to
roll
back
and
downgrade
it's
it's
not
always
it's
not
always
a
just.
Let's
do
it.
B
B
So
I
highlighted
this.
This
particular
sentence
here
because
as
soon
as
I
read
that
sentence
I
knew
immediately
what
the
issue
was
right,
like
you
said
so,
and
you
can
see
it
down
here
right,
node
selector
has
you
know,
node
roll
kubernetes
master,
so
kubernetes
1.24
was
when
kubernetes
removed
the
master
label
and
added
or
changed
it
to
control,
Dash
plane,
and
so
all,
basically,
all
of
this.
Their
whole
outage
happened
because
yes,
they're
using
Calico
and
and
all
this
other
stuff,
but
the
way
that
it
was
set
up
was
it.
B
You
know
in
in
many
different
ways
things
like
schemas
and
apis
and
right
all
of
this
other
stuff
that
can
you
know,
affect
that
cluster.
So
it's
it's
hard
to
roll
back.
You
know
we
looked
at
it
last
week,
doing
the
NCD
restore,
and
you
know
in
that
case
it
was
different
because
we're
trying
to
restore
it
onto
a
completely
different
cluster,
but
you
saw
the
complexity
associated
with
that
and
they
even
walked
through
this
process.
In
this
in
the.
B
Yeah
they
talk
about
how
exactly
like
what
we
did.
You
know
last
week
it's
I
I
break
the
cluster
I,
take
two
control
plane
nodes
down,
I
reform
etcd
as
a
single
node.cd.
To
then
restore
that
state
back
and
then
reform
the
Quorum.
You
know
at
the,
however
many
LCD
nodes
I
have
and
let
it
sink
and
hope
for
the
best.
B
So
it
was
definitely
I.
Don't
know
if
coincidental
serendipitous,
you
know,
take
your
pick,
but
it
was
an
interesting
one.
The
other.
The
other
thing
that
struck
me
as
interesting
is
the
the
deprecation
of
the
that
tag.
Master
on
the
control
plane
notes
was
announced
in
1.20
kubernetes
1.20,
which
was
released
in
December
of
2020.
B
B
Like
you
know,
changing
a
simple
low
label
would
have
that
big
of
an
impact
you
know,
but
on
the
other
hand
you
know
clearly
somebody
some
people
were
thinking
about
it,
because
if
you
look
inside
of
like
openshift
where's,
my
cluster
here,
you
know
so
right
now,
I'm
running
4.12
4.12
is.
Is
it
1.24
based.
B
25.25,
so
so
openshift
is
past
that
that
moment
in
kubernetes
time
frame
where
it
changed.
But
if
I
come
down
here
to
my
nodes,
you'll
notice
that
we
have
both
roles
right,
we
have
both
master
and
control
plane
and
if
I
click
on
my
node
here
and
look
at
yaml
you'll
see
the
there's
labels
for
both
that
are
in
here,
so
control,
plane
and
master.
B
So
clearly,
somebody
inside
of
red
hat
was
thinking
of
this
and
you
know
I,
don't
know
if
we'll
ever
remove
that
I
I
haven't
looked
at
any
of
the
deprecation
notices
or
anything
like
that
for
openshift,
but
it
feels
like
one
of
those
things
of
you
know
hey.
Maybe
we
should
use
this
Reddit
experience
or
learn
from
reddit's
mistakes.
If
you
will
and
take
that
into
account
is
another
thing
but
yeah
it's
one
of
those
things
that
it
just
it
feels
so
small
yeah.
A
And
the
awesome
part
is
that
they
were
so
transparent
about
it.
You
know
I
I,
think
gitlab
a
couple
years
ago
they
had,
they
had
a
big
issue.
Where
I
mean
it,
they
were
down
hard
and
they
they
live
streamed
their
you
know
their
recovery,
and
then
they
wrote
an
awesome
postmortem
like
I,
think
it's
excellent.
When
companies
write
postmortems
like
this,
because
then
you
can
see
to
your
point.
This
was
a
tiny,
tiny
mistake,
right
or
tiny,
tiny
thing
to
overlook
and
the
the
just
the
scaled
effect
that
it
had.
A
B
Well,
on
the
other
hand,
if
you
you
know
again,
if
you
read
through
this
post-mortem,
one
of
the
things
that
they
admit
to
somewhere
up
here
is
like
their
production
and
their
test
environments.
Weren't
the
same
I
mean
they.
They
walk
through
this
whole
thing
of
you
know,
years
ago
they
implemented
and
sorry
I'm
scrolling
randomly
here,
and
so
it's
going
to
help
me
find
where
I
was
going,
but
they
they
implemented
this
production
cluster
years
ago,
and
you
know
they
use
Calico,
they
use
Calico
correctly.
B
They
followed
all
of
calico's
best
practices
done
all
these
things.
You
know,
as
their
cluster
scaled
out,
you
know,
Calico
creates
by
default,
creates
a
mesh
where
every
node
is
connected
to
every
node
and
it
has
to
propagate
those
out.
Well,
as
you
grow
the
cluster
size,
you
know
to
hundreds
of
nodes
that
just
simply
becomes
too
unwieldy
to
manage
and
it
can
affect
performance
and
a
number
of
other
things.
So
the
recommendation
from
Calico,
which
is
exactly
what
they
did,
is
to
set
up
route
reflectors.
So
that
way
the
routing
table
is
shrunk
right.
B
You
can
avoid
those
issues
and
again
they
walk
through
this.
In
here
of
you
know,
yeah
The,
Logical
Choice
was
control,
plane
nodes
because
you're
always
going
to
have
control
plane
nodes
in
a
cluster
that
dynamically
scales
up
and
down.
So
they
should
hold
those
route
reflectors.
So
absolutely
it
makes
sense.
In
that
respect,
the
problem
was
the
original
Ops
Team
is
gone.
You
know
just
natural
attrition
people
taking
other
jobs
I.
B
You
know
all
these
other
things
so
that
the
current
Ops
Team
didn't
know
about
it,
and
it's
just
one
of
those
like
you
know,
production
and
test
didn't
look
like
each
other.
You
know
it
feels
like
one
of
those
I
I
saw
somebody
in
the
comments
somewhere.
They
used
the
term
haunted.
If
you
don't
know
exactly
what's
in
the
cluster,
should
you
treat
it
as
haunted
because
you
don't
know
what
might
happen
and
you
know,
should
you
focus
on?
Should
you
push
on
you
know?
B
Should
she
use
this
as
a
learning
experience
of
no,
we
need
to,
like
authoritatively,
understand,
all
of
our
configuration
again
going
back
to
last
week's
conversation
and
all
the
other
conversations
we've
had
around
disaster
recovery
and
get
offs
and
all
this
so
clusters
should
be
disposable
and
we
should
be
able
to
recreate
them
one
for
one,
not
just
for
disaster
recovery.
But
for
things
like
this,
you
know,
kubernetes
has
so
many
moving
parts,
so
many
components
so
many
things
happening,
an
openshift
adds
a
whole
nother.
B
A
Yeah,
it's
I'm
really
like
Bravo
to
those
guys
because
they
got
through
it.
They
got
everything
back
up
and
running.
You
know
that's,
but
I
I,
really
just
from
a
nerd
perspective
right
like
I
love,
the
transparency
I
think
it's
I
think
it's
awesome
and
it
gives
you
Insight
yeah
and
to
help
people
think
and
how
they're
troubleshooting
and
stuff
like
that
and
like
I
will,
and
they
really
did
they.
A
They
thought
of
almost
everything
right
I
mean
like
they
the
fact
that
they
caught
that
one
little
label
at
the
end
like
that
sucks
that
they
caught
it
late.
You
know
it
would
have
been
cool
if
they
could
have
caught
it
in
the
middle
and
then
restored
or
gotten
up
the
running,
but
yeah
I
think
it's
awesome.
B
Yeah,
it
goes
to
the
you
know
where
they
say
down
here,
like
3.9
billion
lines
of
log
lines
in
in
there,
oh
yeah.
So
it's
a
lot
of
it's
a
lot
of
logs.
You
know
it's
again,
one
of
those
things
that
you
don't
think
about
until
you
have
massive
scale
like
you
know,
I
can
only
imagine
Reddit
right.
It's
a
top
five
Internet
site.
I
think
you
know.
So
how
do
you
accommodate
that
scale?
B
B
So
one
from
what
I
understand
you
know.
You
know
we
had
wire
names
escaping
me
today,
Johnny
it's
GitHub
sky
to
the
Galaxy.
Why
can
I
not
remember
her
name,
Harriet,
no
Hillary
Hillary.
Thank
you,
she's,
going
to
like
send
me
a
murderous
email
and
deservedly
so
Hillary.
So
you
know
Hillary
worked
on
that
team.
You
know
I
think
still
works
adjacent
to
that
team
of
like
everything
is
very
strictly.
You
know
get
Ops
controlled
like
they
have.
B
You
know
very,
very
strict
discipline
about
how
they
interact
with
the
Clusters,
how
they
change
things
in
the
Clusters
and
all
this
other
stuff
and
I
think
that's
part
of
it
is
before
they
update
before
they
touch
any
customer
cluster.
They
have
very
rigorous
testing
plans
that,
and
they
validate
they
verify
that
all
of
their
clusters
match
this
configuration.
If
you
tested
those
configurations,
I
feel
like
I'm
speaking
for
them,
but.
A
It
was,
it
was
interesting
right,
I
mean
like
they
were.
If
you
had
messed
up
like
there
was
like
it
wasn't
just
like.
Oh
my
bad,
it
was
like.
It
was
a
thing
like
you,
you
had
to
sit
down,
you
had
to
explain
what
you
had
to
do
or
what
you
did
and
why
you
did
it.
You
know
I
mean
it's.
It's
serious,
yeah.
B
Of
course,
we've
talked
about
here
on
the
live
stream,
where
you're
putting
all
of
the
control
plane
Services
all
those
pods
that
make
up
the
control,
plane,
you're
hosting
them
in
a
different
cluster,
with
the
compute
nodes,
the
worker
nodes
for
the
cluster
being
separate,
so
that'll
certainly
change
how
all
of
that
works
and
how
they
control
and
manage
those
things.
I
haven't
fully
digested
how
that
impacts,
Disaster
Recovery
in
theory.
B
In
theory,
your
control
plane,
is
now
it's
a
set
of
pods,
which
it
always
has
been
but
they're
using
things
like
PVCs.
You
know,
ncd's
data
is
now
stored
on
a
PVC
and
it's
abstracted
from
the
underlying
cluster.
So
if
you're
hosting
cluster
were
to
be
affected,
you
can
simply
restore
those
pods
to
a
different
cluster
redeploy
them
on
a
different
cluster
with
the
same
PVC
data,
and
it
would
come
back.
You
know
as
long
as
you
can
point
the
route
back
at
them,
so.
B
We
did,
we
were
talking
about
ansible
wisdom
and
Craig.
Was
the
person
doing
the
presentation
during
the
keynote
he
was
talking
about?
How
chat
gbt
is
really
really
awesome,
but
it
doesn't
know
when
it's
wrong,
so
it
it
has
these
very
authoritative
sounding
matter
of
fact
like
this
is
the
right
thing,
and
it
has
no
clue
that.
No,
that
that
isn't
right
at
all
and
and
it's
funny
because
I've
seen
a
number
of
internal
messages
recently
about
like
hey
I,
was
trying.
You
know.
B
I
asked
chat
GPT
this
and
it
said
that
I
could
do
this,
and
one
of
them
was
like,
including
some
security
information
or
something
like
that
in
the
install
config.yaml.
It's
like
no
clearly
that,
like
that,
isn't
there
but
chat
gbt
was
like
oh
yeah,
absolutely
drop
this
yaml
in
and
install
config
and
it'll
deploy
these
things
and
you're
clustered
you're
good
to
go
and
I'm
like
oh,
oh
yeah,
so.
B
I
I
do
you
know
definitely
having
an
inkling
of
an
idea,
we're
going
to
be
showing
during
the
Keynotes
and
stuff
at
Summit
this
year,
keep
an
eye
on
what's
happening
with
wisdom,
there's
some
really
cool
stuff,
that's
happening
outside
of
there
and
I
I
hope
to
someday
that
we
can
have
the
wisdom
folks
on
here,
so
that
we
can
talk
about
a
lot
of
the
behind
the
scenes
stuff
and
how
they
operate
and
how
they
train
their
models
and
all
this
other
stuff,
because
there's
some
really
awesome
stuff.
There.
B
B
Well,
if
you're
not
familiar
ansible
wisdom
is
so
we
announced
it
back
at
ansible
fest
last
year.
There's
some
videos
that
are
available
on
I
think
the
ansible
YouTube
channel
right
now
effectively.
It
is
a
a
machine,
learning
AI
thing
that
allows
you
to
quickly
and
easily
create
ansible
playbooks,
so
the
demos
that
that
they
show
are
effectively
you
go
and
you
create
a
new
playbook.
You
know
using
your
IDE
editor
and
you
you
create.
B
Basically
you
give
it
a
name,
create
a
new
virtual
machine
in
Azure
and
the
the
machine
learning
model
basically
spits
out
the
ansible
Playbook,
the
ansible
yaml
that
you
need
in
order
to
do
that,
and
then
you
know.
Okay
me,
the
human
I
review
it
I
make
sure
that
it
is.
You
know
it
has
what
I
need
into
it.
So
next
step
is
now
create
a
new
Azure.
Network
now
connect
my
VM.
B
You
know-
and
you
just
step
through
this-
basically
naming
your
steps
and
it
takes
what
it
infers
from
the
name
to
actually
make
that
happen.
So
it's
really
cool
there's
a
lot
of
really
cool
things
that
I
think
can
happen
there.
With
the
with
that
wisdom,
the
the
ml
model,
like
I,
said,
keep
an
eye
on
Summit
and
the
Keynotes
that
we'll
have
there
all
right.
B
Ops
is
fun
to
me
and
seeing
what
happens
inside
of
one
of
the
biggest
websites-
and
you
know
literally
in
the
world-
is
always
interesting
to
me.
Yeah
same
oh,
we'll
I'll,
quickly,
I
said
that
with
the
last
one,
I'll
quickly
cover
this
one,
because
I
got
the
tab
open.
So
somebody
asked
this
week
about
a
I
think
they
misinterpreted
this
segment
of
the
documentation.
Let
me
let
me
post
this.
B
When
you
look
at
this,
what
it
says
is,
if
you're
using
vxlan,
then
there
are
a
number
of
reasons
that
effectively
pod
to
pod
Communications
become
limited
to
approximately
one
gigabit
per
second.
So,
even
though
the
host
may
have
10
or
40
or
100
gigabit,
whatever
networking
PODS
of
pod
Communications
get
limited
by
the
tunnel
by
the
encapsulation
and
various
things
that
happen
inside
of
there
I
think
one
of
the
vacuum
trucks
is
going
through.
The
neighborhood
I
hear
this
very
loud
buzzing.
I,
don't
know
if
it's
coming
through
the
microphone,
but
I.
A
B
Hear
it
yeah
they,
the
city,
sends
you
know:
people
dump
their
yard
grass
streaming
all
that
on
the
no
next
to
the
road
and
they
come
through
and
vacuum
it
up,
anyways,
so
yeah
anyways,
the
end
result
is
yes
at
least.
It
appears
according
to
the
documentation
here
that,
if
you're
using
vxlan,
which
openshift
sdn
uses
vxlan
pod
deck
pod
Communications
might
be
limited
to
one
gigabit
per
second.
B
That
being
said,
I've
asked
for
some
clarification
and
some
additional
details
on
this,
because
I've
also
seen
performance
testing
that
shows
far
more
than
one
gigabit
being
possible.
So
I
don't
know
if
this
is
just
based
on
outdated
information.
I
mean
here,
let's
roll
all
the
way
back
well
page
doesn't
exist,
okay.
So
how
far
back
does
this
page
go
yeah?
So
at
least
in
4.4,
which
4.4
keep
in
mind
was
two
and
a
half
plus
years
ago,
three
plus
years
ago,
Like
A
lot's
changed.
B
So
you
know.
Maybe
something
is
different
here.
Maybe
there's
something
something
else,
but
I've
asked
for
clarification
here.
If
you
think
that
this
is
an
issue
or
if
you
think
that
you're
encountering
this
there's
a
couple
of
different
ways,
you
can
work
around
it
for
lack
of
a
better
term.
One
is
to
use
multis
and
connect
the
Pod
to
a
secondary
Network.
You
know
one
that
isn't
you
know,
maybe
it's
directly
connected
to
the
outside
world
or
something
like
that.
B
Migrating
from
the
openshift
sdn,
so
you
can
see
here
that
it
walks
you
through
exactly
how
to
upgrade.
How
did
well
I
say
upgrade
how
to
transition
I,
don't
know
if
it's
an
upgrade.
It
depends
on
your
perspective,
I
guess,
and
this
has
been
in
place
for
a
long
time.
Maybe
four
eight
yeah
I
think
four
eight
was
the
first
version
that
OV
and
kubernetes
was
fully
supported,
so
it
it's
been
in
there
for
a
long
time.
So
you
can
absolutely
do
that.
B
Update
if
you
so
choose
the
other
one
would
be
to
use
a
third
party.
You
know
a
different
sdn.
You
know
a
Cisco
ACI,
you
know
uses
I,
don't
know
if
that
it
uses
an
encapsulation
so
much
as
it
just
connects
directly
to
the
Upstream
switches
and
does
whatever
it
does.
Calico.
Of
course,
Calico
I,
don't
think
I
think
it
operates
at
layer
three
and
uses
routing
instead
of
encapsulation
there's
a
number
of
others
that
are
out
there
that
use
all
different
protocols.
So.
B
B
Let
me
share
this
page
with
the
documentation
here
in
chat,
so
this
is
I
think
we
said
before
this
is
a
a
feature
capability
that
was
added
in
4.12..
There's
a
couple
of
interesting
or
a
couple
of
important
things.
I
should
say
to
know
about
it:
one.
It
is
only
supported
with
a
subset
of
cloud
providers,
so
you
can
see
AWS,
Azure
and
vsphere.
B
The
other
thing
to
note
here
is
only
with
AWS.
Is
it
activated
by
default,
meaning
as
soon
as
you
deploy
the
cluster
right
as
soon
as
openshift
install
is
done
running
only
with
AWS?
Do
you
have
a
control,
plane
machine
set
in
place
with
Azure
and
vsphere
you?
The
administrator
have
to
go
in
and
basically
turn
it
on.
You
have
to
create
the
an
instance
of
the
CR
in
the
custom
resource,
so
you
can
see
this
docs
page
kind
of
goes
through
and
explains
all
of
that.
B
B
So
what
I
wanted
to
go
through
today
is
going
through
this.
You
know:
managing
control,
plane
machines
with
the
machine
set
operator
and
there's
a
couple
of
different
things
that
I
wanted
to
look
at.
So
the
first
one
and
I
didn't
I,
actually
didn't
notice.
This
Johnny
pointed
this
out,
and
that
is
the
rollout
strategy
and
I,
don't
know
where
it's
at
anyways
we'll
look
at
it
in
the
ammo
in
just
a
moment
yeah.
B
So
what
I
think
happens
is
so
with
a
traditional
machine
set
if
I
go
and
make
a
modification
to
that
machine
set,
nothing
happens
until
I
deploy
a
new
machine
from
that
machine
set,
and
then
it
will
have
whatever
the
new
config
is.
It
doesn't
go
back
and
retroactively
change
the
configuration
of
the
existing
machines.
B
So,
let's
pick
on
you
know
here.
I'll
come
over
to
my
cluster
here.
So
in
my
cluster
here
I've
got
a
number
of
different
machines
and
from
a
compute
perspective,
I
only
have
one
machine
set.
You
can
see
it's
in
Us,
East,
2C
and
if
I
were
to
come
in
here
and,
for
example,
modify
this
instance
type,
it
doesn't
change
anything
until
I
deploy
a
new
machine
and
then
it
would
use
that
new
machines
instance
type.
B
B
So
the
first
thing
that
we
want
to
do
is
come
down
here
to
our
custom
resource
definitions,
so
you'll
notice
here
in
compute
there
is
no
navigation
item,
I,
guess,
yeah
menu
item
here
for
control,
plane
machine
sets,
the
crd
we
find
by
searching
here
for
control
plane,
and
we
have
this
control
plane.
Machine
set
and
if
I
switch
over
to
instances
I
have
one
control
plane
machine
set
instance
named
cluster
and
if
I
browse
our
yaml
here,
there's
some
interesting
things
that
we
can
see
about
this
particular
object.
B
So
one
three
replicas.
Of
course,
you
know
we
want
to
have
three
control
plane
nodes.
So
here's
that
update
strategy
so
type
rolling,
update,
so
I'm
thinking
and
we're
going
to
find
out
in
a
minute
that
this
will
cause
it
to
automatically
trigger
a
a
trigger,
a
redeployment
in
the
event
of
a
change.
B
B
A
And
I'm
just
going
to
answer
a
question
real
quick,
so
that
way
it
gets
across,
but
babuni
had
asked.
Can
we
scale
up
the
the
control
plans
from
say
three
to
five
using
the
control
plane
machine
set
and
the
in
the
docs?
It
clearly
says
that
horizontal
scaling
of
the
control
play
machine
set
is
not
supported.
So
it's
still
a
maximum
of
three
control
plane,
nodes,
yeah.
B
Yeah,
you
know
red
hat
I,
don't
know.
Surely
somewhere
there's
like
there,
someone
has
a
support
exception
for
more
than
three
I
I
have
to
guess,
but
by
default
you
know
we
only
and
always
only
support
three
control
play
nodes
unless
it's
a
single
node,
openshift
and
then
there's
one
of
course,
but
yeah,
it's
I
I,
know
I
am
sure
you
could
force
it
to
happen.
You
know
you
could
go
in
and
you
could
modify
all
of
the
appropriate
things.
It
may
not
even
be
that
hard.
B
It
may
even
be
just
changing
the
number
and
it
does
it
itself.
It
just
goes
yeah,
but
technically
that
still
wouldn't
be
supported
right.
We
we
only
support
three
I
I'm
curious
I
should
go
and
look
at
the
road
map,
because
I've
heard
Rumblings
of
various
things
that
don't
change
the
number
of
SCD
nodes
but
affect
other
things
when
it
comes
to
control
plane,
so
I
feel
like
I
feel
like
we
should
probably
have
that
product
manager
come
on
and
talk
about
what
might
be
happening
there.
B
Yeah
thanks
for
the
question,
so
failure
domains,
AWS
placement
and
availability,
no
availability
Zone
if
I
could
speak
Us,
East,
2C
and
then
from
here
it
just
kind
of
filters
down.
You
know
here
tells
us
what
what
network
it's
going
to
use
that
type
of
stuff
and
then,
after
that,
it
goes
into
the
just
like
any
other
machine
set
kind
of
the
labels
and
the
other
things
that
it's
going
to
assign
to
our
deployed
notes.
B
So
you
can
see
here
the
spec
and
here's
the
instance
type
that
it's
going
to
use
and
the
type
of
block
device.
You
know
the
type
of
storage
it's
going
to
use
for
the
virtual
machine
and
all
the
normal.
You
know
AWS
stuff,
that
it
needs
for
security
groups,
and
you
know
all
that
stuff,
so
all
the
stuff
that
we
need
in
here
controls
via
machine
set
just
like
with
any
other
thing.
So
what
I
want
to
test
out
here
is
what
happens
if
I
take
this
and
I
duplicate
it
for
2B.
B
B
A
B
So
yeah
so
to
be
clear,
this
cluster
was
provisioned
in
a
different
way
than
I,
normally
provisioned
them.
This
is
not
a
default.
Ipi
install
this
one
was
customized
by
the
rhpds
folks,
so
they
limited
the
cluster
just
to
2C
and
I,
wonder
if
that
will
have
any
impact
on
us.
So,
let's
find
out,
let's
click
save
so
in
theory.
B
If
that
roll
out
strategy
that
we
were
looking
at
up
here
before
where'd,
you
go
anyways
it
if
it
is
set
to
or
if
it
will
take
automatic
action,
then
maybe
what
we're
going
to
see
here
in
a
moment
is
our
machines
being
changed,
so
it
has
already
failed
one
of
the
control
plate.
Nodes
and
I
can
only
assume
that
it's
not
working
yeah,
so
control,
plane,
control,
plane,
worker
worker
control
plane,
so
it
hasn't
removed
the
node
from
the
cluster
yet,
but
the
machine
has
been
failed.
A
B
Yeah
it
does
868
versus
Zed
eight
Charlie,
so
yeah
that'll
be:
let's
see
what
happens
here.
Let's
see
events
where
you
failed,
no
subnet
IDs
were
found.
You
know
that
explains
that,
so
our
networking
trick
did
not
work
here.
Let's
go
back
and
let's
just
see,
I,
don't
think
you
can
use
the
same.
Subnet
you've
got
a
cluster.
You
can
try
this
though
right
yeah.
B
B
B
Right,
save
reload,
okay,
so
usds2c,
where
we
know
we
have
a
subnet
is
going
to
use
our
existing
subnet
appear
in
machines,
zero,
one,
one:
okay,
so
it
hasn't
changed.
Anything
up
here
delete
delete
all
right,
failed
nope.
So
that's
not
going
to
work
yeah
same
thing:
no
subnet,
IDs,
okay,
so
that's
interesting
seems
like
they're,
maybe
I'm
sure
that
there's
a
step
that
I'm
missing
here,
which
is,
namely
creating
a
subnet
on
that
particular
availability,
Zone,
so
I'm
looking
at
over
here,
is
API
machine
set.
B
B
Interesting
rough
Edge
to
encounter,
we
should
definitely
double
check
with
the
PMs
to
see
if
there's
an
open,
rfe
or
something
like
that
or
maybe
I'm
just
missing
something
in
there.
But
you
want
to
take
over
the
screen
share,
just
to
show
the
changing
like
this
instant
size,
yeah
well,
I'm,
just
like.
B
A
Yeah,
no
such
luck
yeah,
so
all
I'm
going
to
do
here
is
just
change
the
machine
type.
So
we
we
actually
get
this
question
all
the
time
right,
Andrew,
it's
like
yeah.
How
do
I,
how
do
I
change
the
machine
size
or
how
do
I
change
the
type
of
my
control,
plane,
node
and
so
I'm
just
going
to
go
in
here
and
I'm
going
to
make
that
deal
2XL
from
an
x-large
and
then
I
need
to
move
my
monitor
so
I
can
see
my
buttons
and
save
and
reload.
B
B
So
if
your
node
fails,
for
whatever
reason
it
will
go
in
it
will
redeploy
right,
it'll
create
a
new
node
and
rejoin
it
as
a
control,
plane
node.
So
it's
super
helpful
for
those
scenarios
where
something
has
happened.
Something
has
broken
and
I
need
to.
You
know:
I
I
need
to
reprotect
my
my
control
plane.
It's
literally
going
in,
and
just
you
don't
even
have
to
do
anything
all
right
delete
the
machine
at
worst
and
it'll,
recreate
it,
which
you
saw
with
that
failed
one
that
I
that
I
did
right.
I
it
was
failed.
B
A
B
A
A
All
right,
so,
let's
go
back
over
here
to
machines.
I
just
wanted
to
point
that
out.
Then
the
same
thing
goes
back
to
like.
If
you
wanted
to
change
your
volume
size
or
or
even
like
your
type
or
whatever,
like
all
the
things
that
you
would
do
in
a
normal
machine
config,
you
can
just
go
in
here
and
make
that
change,
and
then
it
will.
You
know,
it'll
make
the
chain
it'll,
make
the
change
and
then
redistribute
and
then
to
Andrew's
Point
earlier
about
the
the
update
type.
B
B
B
Vms
in
vsphere,
so
I
can
see
how,
depending
on
how
your
Zone
configuration
is
inside
of
that
Visa
environment,
that
you're,
using
again
Tech
preview
and
412,
but
you're
using
that
zone,
feature
how
you
could
basically
mimic
that
type
of
setup,
at
least
at
deployment
time,
because
there's
nothing
that
forces
you
to
have
those
Zone
definitions,
I,
don't
think
on
like
a
data
center
or
a
DRS
cluster
I
think
you
can
put
them
on
individual
hosts,
so
you
could
force
it.
Maybe
that
way
to
deploy
across
multiple
hosts.
B
A
B
Do
keep
in
mind
one
thing
just
from
a
pragmatic
approach,
if
you're
going
back
to
the
NCD
performance
thing,
if
your
NCD
is
kind
of
at
the
threshold
of
performance,
you
know
where
it's
it's
already
having
issues,
you
know
read,
write
Etc.
If
you
aren't
careful,
you
know
cognizant
about
the
time
that
you're
doing
that
it
can
cause
extra
stress
on
that.
So
what
I
mean
by
that
is
I've
got
my
three
at
CD
nodes
or
my
three
control
plane.
B
Nodes
SCD
is
having
some
latency
issues,
and
so
oh
no
I
need
to
fix
this.
Let
me
go
in
and
let
me
change
the
data
store
that
it's
using
and
it
begins
redeploying
those
that
CD
nodes
or
a
control
play
nodes
when
it
does
that
the
existing
SCD
nodes
have
to
populate
the
new
etcd,
which
is
going
to
cause
a
a
influx
or
a
surge
and
read
activity
from
that
Source
at
CD
or
those
Source
at
CD
nodes.
So
just
keep
in
mind
that
you
can
inadvertently
affect
things
like
that.
You
know.
B
A
B
All
right,
so
any
questions
around
that
you
know
feel
free
to
ask
those
in
the
in
the
chat
here.
It's
again,
we
keep
adding
these
things.
That,
on
the
surface,
are
amazingly
simple
right.
You
went
in
and
you
you
changed
one
thing,
and
now
it's
redeploying
the
control
plane
like
which
is
a
massively
complex
thing,
as
we
saw
last
week
of,
and
we
also
demoed
before
you
know
how
to
replace
a
failed
control,
plane
node.
B
A
B
A
I
agree
man,
I
I,
just
messing
with
it
this
morning
and
looking
at
it
I'm
like
dude,
it's
so
easy.
You
know
and
I
think
that
that's
that
that's
where
a
lot
of
times
we
kind
of
like
I,
don't
want
to
say
it
for
everything,
but
we
missed
the
mark,
sometimes
right
where
it's
like.
We
make
this
really
awesome
thing,
but
we
make
it
kind
of
difficult
to
use.
You
know
this
is
just
straight
up
simple.
You
know
like
hey
we've
already,
if
you're
in
AWS
we've
already
set
it
up
for
you,
here's
the
CR!
B
B
B
Right
so
the
second
thing
we
wanted
to
talk
about
today
is
infrastructure
notes,
so
I'll
reshare,
this
guy,
so
I
did
go
in
here.
Just
to
show
everybody
I
just
went
in
and
removed.
You
know
I'm
looking
at
a
regular
machine
set
here
we
go
so
I
just
went
in
and
removed
the
stuff
that
we
added
so
we're
back
to
back
to
normal
on
this
one
and
you'll
notice
that
it
also
as
soon
as
I
did
that
it
also
cleaned
up
that
failed
machine.
B
So
that's
kind
of
cool
anyways,
so
we're
going
to
talk
about
infra
nodes,
so
infra
nodes
are
something
that
we're
first
I.
Don't
I,
don't
know
that
other
kubernetes
is
had
the
concept
of
an
infra
node
before
openshifts,
but
I
first
became
familiar
with
them
back
in
the
openshift
three
days,
openshift
three
I
believe
they
were
required.
Yes,
you
always
had
control,
plane,
infra
and
you
know
computer
worker
nodes
with
openshift
4.
B
They
became
optional
and
infra
nodes
are
useful
for
a
number
of
different
reasons,
not
the
least
of
which
I
always
like
to
point
out
is
the
entitlements
for
them
are
included.
So
just
like
with
control,
plane
nodes,
you
don't
have
to
use
any
of
your
entitlements
with
infra
nodes
or
con
or
control
play
nodes
only
for
your
worker
nodes,
which
means
that
it's
a
great
way,
particularly
once
you
start
adding.
You
know
a
layered
features:
functionality
of
openshifts
to
save
on
those
resources
so,
for
example,
logging
service
metric
service
right.
B
B
B
So
one
if
we
look
in
the
documentation
here,
it
will
help
us
walk
through
some
of
those
things
that
qualify
following
infrastructure.
Workloads.
Do
not
re,
do
not
need
or
do
not
require
worker
subscriptions.
So
you
can
see
the
router
Ingress
right.
The
integrated
registry
see
Quay
odf
ACM
ACS
openshift
get
Ops
pipelines,
there's
also
in
the
subscription
guide.
B
That
link,
so
the
subscription
guide
also
details
a
number
of
things
that
Encompass
or
infra
nodes,
if
I
scroll
all
the
way
down
here
by
the
way
we're
changing
the
name
of
this
assuming
my
change
request
gets
approved,
we're
removing
sizing
from
the
title
of
this,
because
this
doesn't
talk
about
the
sizing
of
an
openshift
cluster.
Rather,
it
gives
an
example
of
the
the
size
quote:
unquote
of
your
entitlement
pool
based
on
your
cluster
layout.
So
in
order
to
avoid
confusion,
I
put
in
a
request
that
we
change
or
remove
sizing
from
it.
A
B
Usually
not
I'll
I
say
that,
because
here
we
go
infrastructure,
notes,
I
say
that,
because
let's
pick
on
I,
don't
know
a
pick
on
some
third
party.
You
know
one
of
our
partners,
who
provides
say
a
registry,
so
the
integrated
openshift
registry,
as
well
as
Quay,
can
both
be
hosted
as
infrastructure
workloads,
but
a
third
party,
you
know
even
our
partners,
their
registry
product
would
not
be
considered
a
part
of
that
infra
workload.
B
B
The
the
definition
is
a
little
bit
fuzzy
for
my
taste.
You
know
I'm
a
very
precise
language
person,
but
you
can
always
ask
your
account
team.
They
can
there's
ways
that
they
have
escalating
back
to
the
bu
back
to
the
folks
who
manage
these
things
to
get
confirmation.
If
you
have
any
questions
about
it,
so
you
can
see
here.
Examples
of
non-red
hat
software
that
qualifies
infrastructure
include
custom
and
third
party
I'll,
highlight
here
monitoring
agents
so
again
to
use
an
example
from
our
partner
ecosystem
datadog.
B
The
datadog
agent
can
be
deployed
across
infra
and
control
plane
nodes
and
it
doesn't
disqualify
them
from
you
know
having
the
entitlement
exception.
You
know:
CSI
drivers,
cni
interfaces,
hardware,
virtualization,
enablement
accelerators,
which
is
a
weird
phrase.
I'm
pretty
sure.
That's
referring
to
things
like
the
Nvidia
operator
and.
B
You
know
the
the
stuff
that
it
needs
for
managing
gpus
and
all
that
controller
pods
used
for
kubernetes
here
to
your
operators,
custom
or
third-party
software.
So
this
is
going
back
to
like
NetApp
Trident.
So
tridents
is
a
both
an
operator
that
it
uses
to
manage
itself
as
well
as
a
a
demon
set
that
deploys
pods
across
all
the
nodes,
so
kind
of
those
two
things
mean
that
it
doesn't
or
it
is
qualified
as
an
infrastructure
workload.
So
yeah
I
would
say.
A
To
say
he
asked
about
like
an
nginx
Ingress
controller
as
an
example
I'm.
In
my
mind,
I'm,
like
that
makes
sense
that
that
would
be
on
the
info
node
as
a
controller,
but
yeah
definitely
definitely
definitely
reach
out
to
your
account
team
and
let
them
go
fight.
The
fight
for
you
make
sure
that
it's
supported
yeah
for
you,
yeah.
B
And
it's
there's
other
things
like:
let's
see,
there
is
one
down
here
about
Rel
entitlements
as
well
like
yeah,
red
Enterprise,
Linux
entitlements
for
supporting
services.
So
in
the
case
of
so
let's
say
you
deploy
UPI,
you
want
to
use
h,
a
proxy
or
or
nginx
or
whatever
deployed
onto
a
Rel
VM.
You
know
or
Rel
node,
as
the
load
balancer
that
sends
traffic
into
your
cluster.
Well,
that
isn't
included
in
those
in
openshift
entitlements
either.
B
So
if
I'm
deploying
vsphere
UPI
I
cannot
use
openshift
entitlements
for
a
another
VM
hosted
in
vsphere
to
act
as
the
load
balancer,
you
know,
or
host
or
proxy
nginx
whatever
for
my
cluster,
but
if
I'm
deploying
into
openshift
virtualization
that
Rel
entitlement
is
included
so
slightly
confusing
and
that
openshift
virtualization
thing
I
think
they
use.
There's
a
blurb.
B
Quay
data
Foundation
there's
a
blurb
way
up
here
about
yeah
here.
This
includes
the
entitlement
to
use
red
hat
Enterprise
Linux,
as
the
guest
operating
system
in
the
virtual
machines
hosted
on
openshift.
So
it's
it's
very
explicit
about
for
VMS
hosted
in
openshift
using
openshift
virtualization
Rel
entitlements
are
included,
which
is
really
cool.
That.
A
B
B
Know
how
do
subscriptions
work?
When
do
you
Auto
scaling
in
spot
instances?
I,
don't
know
how
often
it
is
adjusted.
My
understanding
is
that
the
pool
should
be
large
enough
to
accommodate
your
surge
capacity,
but
clearly
we
do
on-demand
licensing
or
entitlements
rather
for
things
like
you
know,
Rosa
and
and
Aro,
so
I
I
would
definitely
ask
your
account
team
about
that.
Waleed
of
whether
or
not
there
is
a
way
that
you
can
do
sort
of
on-demand
entitlement
pools
for
you
know
your
on-prem
clusters
stuff,
like
that.
A
A
Yes,
on
the
disconnected
side
of
this
I
didn't
want
to
say
it
but
yeah.
It
is
talk
to
your
account
team,
though
right,
because
they
they
do
have
ways
to
account
for
for
bursting
right,
because
we
back
in
back
in
the
day.
You
know
we
had
a
customer
that
was
using
AWS
and
they
had
similar
concerns.
Like
you
know,
I'm
I'm
gonna
be
I'll
run.
A
B
B
So,
if
the
underlying
the
physical
openshift
cluster
is
only
running
tenant
or
hosted
openshift
clusters,
you
don't
have
to
entitle
that
cluster
if
you
run
other
workloads.
So
if
you're
deploying
other
non-openshift
VMS
in
there,
if
you're
deploying
other,
you
know,
containers
pods
that
aren't
you
know
they're
doing
other
things,
then
you
would
have
to
entitle
using
socket
entitlements
those
nodes.
B
So
you
entitle
in
that
scenario.
You
would
entitle
the
cores
for
the
virtual
clusters.
So
there
is
some
I've
seen
some
chatter
about
that,
and
you
know
narrowing
those
rules
down,
particularly
in
light
of
hosted
control
planes.
So
they
are
working
on
a
hosted
control,
plane
provider
for
openshift
virtualization
and
that
might
change
that
calculus
a
little
bit
so
I'm,
hoping
that
we
can
learn
more
as
we
get
closer
to
that
being
a
reality.
But
that's
that's
later
this
year
before
we'll
really
know
any
more
about
that.
B
B
Don't
think
so
because
they're
managed
by
keep
alive
D
and
the
API
IP
will
always
be
on
a
control,
plane,
node
and
the
Ingress
or
startout
apps
IP
will
always
be
on
a
node
that
hosts
an
Ingress
controller,
which
is
usually
going
to
be
either
a
standard
worker
or
an
infra
node,
but
it
could
be
a
control
play
node
as
well.
Yeah
there's!
No.
If
you
look
at
that,
there's
no
magic!
That's
happening
there.
It's
not
like
a
traditional
load
balancer
where
you
can
say.
B
B
On
any
platform,
so
it
gives
somewhere
in
here,
I
think
I.
Remember
there
being
a
sample
Come
on
load,
balancer
config!
No,
did
we
pull
it
out
there?
It
is
so
in
here.
It
basically
says
you
know
if
it
comes
in
on
Port,
6443
always
send
it
to
these
these
nodes.
If
it
comes
in
on
22
623,
send
it
to
these
nodes.
B
443
send
it
to
these
nodes,
there's
no,
nothing,
no
actual
load
balancer
behind
those,
it's
just
a
virtual
IP
that
is
hosted
on
a
node
that
has
the
right
Ingress
for
it
to
be
able
to
access
that
traffic
type
and
that's
just
how
it
how
it
functions
so
I
wish
it
was
a
little
smarter.
It
would
make
a
lot
of
things
easier.
B
Okay,
so
keep
questions
coming
in
y'all.
Thank
you.
So,
let's
switch
back
over
to
our
infrastructure
stuff
over
here,
so
creating
machine
sets
is
really
straightforward.
So
I'll
point
out
two
different
things
here,
so
one
you
can
create
these
using
if
you're
on
a
infrastructure
that
has
machine
API
integration,
you
can
create
machine
sets
in
order
to
manage
your
infra
nodes,
just
like
with
compute
nodes.
B
B
So
we
use
a
taint
to
repel
other
workload
from
that
node
and
then
we
have
to
go
in
and
that's
why
there's
this
whole
section
in
the
documentation
here
about
we
have
to
go
in
and
we
have
to
add
the
Toleration
to
the
specific
workload
that
we
want
to
go
on
to
that
node.
So
it
walks
through
down
here-
and
it
goes,
you
know:
here's
how
to
move
the
router.
B
You
know
the
Ingress
controller
over
to
our
infrastructure
node,
so
I
want
to
quickly
attempt
to
create
an
infrastructure
machine
set
here
and
I
thought
that
there
was
and
there's
compute
machine
sets.
I
thought
there
was
one
that,
like
explicitly
here's
how
to
do
it
on
AWS
but
I
guess
not
yeah,
here's
vsphere!
A
A
B
These
things,
Etc
and
yeah
I
know
I
need
to
change
a
bit
of
a
bunch
of
other
things
in
here
too.
All
of
that
can
stay.
All
of
that
can
stay
sure,
we'll
keep
all
the
tags
and
we
can
get
rid
of
the
status
right
so
now,
I
think
we're
down
to
the
minimum
or
the
the
actual
fields
that
we
need
here.
So
the
first
thing
we
want
to
do
is
change
this.
Basically
any
instance
of
the
word
worker
to
infra.
B
B
So
we
now
have
our
two
machine
sets
that
are
running
inside
of
here.
Hopefully,
if
we
come
over
here
to
machines,
we
see
our
two
new
machines
that
are
getting
created
and
if
I
come
here
to
the
machines,
look
at
the
ammo,
so
you
can
see
here's
our
labels
eventually,
once
they
get
provisioned
and
join
the
cluster
they'll
show
up
as
nodes
that
will
have
an
appropriate
label
or
a
role
assigned
to
them
inside
of
here.
B
So
there's
some
debate.
You
know
it's
kind
of
a
preference
thing
on
whether
or
not
you
want
to
have
a
separate
machine.
Config
fool
wow
that
was
a
I.
Just
I
cannot
talk
machine.
Can
pig
pool,
config
pool
for
your
infrastructure
nodes
I
am
of
the
opinion
that
you
do
want
to
have
a
separate
machine,
config
pool
for
infra
notes.
I
say
that
for
two
different
reasons,
one,
there
is
probably
some
specific
config
that
you
want
to
use
on
those.
Maybe
that's
a
different
set
of
SSH
keys.
So
that's
you
know.
B
B
So
when
I
go
and
trigger
a
cluster
update,
it
will
roll
that
out
to
all
of
the
machine
config
pools
simultaneously.
So
that
means
that
I
can
do
things
like
pause,
just
the
infinodes
or
just
let
the
infrared
nodes
update.
You
know,
control,
plane
and
infra
update,
for
example,
if
I'm
doing
a
fast
forward,
while
letting
the
compute
nodes
do
their
thing,
whether
that's
to
have
them
be
paused
or
have
them
to
move
forward.
B
The
other
thing
is
with
the
machine
set,
we
can
scale
those
independently,
so
you
may
end
up
in
a
scenario
where
you
have
maybe
a
bunch
of
infrared
nodes
or,
conversely,
a
few
infra
nodes
and
a
lot
of
compute
nodes
and
if
they're,
all
in
one
pool
and
you're
doing
one
up.
You
know
one
node
update
at
a
time
which
is
the
default
for
a
machine
config
pool.
It
can
take
a
very
long
time
comparatively,
and
maybe
you
want
that
to
happen
faster.
B
A
B
B
B
B
B
A
B
A
B
And
now
we
got
two
more
coming
in
all
right
anyway,
so
that
way
our
taint
will
automatically
be
applied.
You
know
when
the
machines
get
provisioned,
we
don't
have
to
worry
about
going
in
and
adding
it
in
later.
You
can
add
it
in
later
just
be
aware
that,
as
you
modify
the
machines,
as
we
just
saw,
it,
wouldn't
be
reapplied
without
having
it
in
the
machine
set
so
we'll.
B
B
So
it's
perfectly
valid.
For
example,
like
hey
I
know,
I've
got
this
application
that
has
just
incredibly
massive
amounts
of
Ingress,
that's
happening
or
even
Ingress,
because
you
can,
you
know,
use
egress
IPS
and
you
can
put
an
egress
IP
onto
an
infinode,
so
my
application
does
all
of
its
Ingress
egress
and
there's
huge
amounts
of
this
and
I
want
to
pin
that
to
a
specific
node
type.
So
you
can
absolutely
create,
you
know,
create
your
node,
give
it
the
infra
label,
give
it
another.
You
know
node
label,
node
annotation,
something
like
that.
B
That
says
you
know
application
y
Ingress
or
application
y
network
or
whatever,
and
then
you
know
effectively
take
your
Ingress
controllers
and
just
like
we
see
here,
go
through
and
modify
it
come
on,
go
through
and
modify
it
so
that
your
Ingress
controllers
for
that
application,
land
on
that
node
and
nothing
else-
does
right,
use
a
node
selector.
You
can
even
add
an
additional
taint
if
you
wanted
to
to
further.
B
B
For
this
you
know,
infra
equals
reserved,
no
execute
and
then
another
one.
For
you
know
a
taint
called.
You
know:
application
X,
you
know
dot,
Ingress
dot,
you
know
whatever
slash,
you
know,
don't
use
and
then
down
here
on
your
router.
You
would
have
both
of
those
tolerations
as
well
as
a
label
or
a
node
selector
and
now
I've
gone
I
have
a
standard,
Ingress
node
or
a
standard
infra
node
I
have
an
Ingress
infranode
and
my
Ingress
infranode
only
has
the
the
routers.
The
Ingress
controller
is
hosted
on
it.
B
A
B
And
memory,
so
different
sizing,
you
know,
has
both
a
taint
for
both
infra
as
well
as
logging,
my
logging
service,
and
we
can
see
like
down
here,
moving
openshift
logging
resources.
I,
add
in
you,
know,
taints
or
excuse
me
tolerations.
You
can
see
this
one
has
two
tolerations,
so
I
add
in
those
tolerations
as
well
as
a
node
selector,
which
you
can
see
here.
We
have
a
node
selector
here
and
I
I've
now
pinned
specific
workloads
on
there,
so
you
can
have
kind
of
this
generic
infra
one.
B
You
know
let
all
of
my
services
that
I
don't
care
about
or
I
don't
want
a
specific
or
dedicated
node
type.
They
just
generically,
select
infra
and
then
for
my
specific
ones
that
I
want
to
have
on
or
my
specific
services
that
I
want
on
specific
nodes.
I
have
both
an
additional
node
selector,
as
well
as
additional
tolerations
to
allow
them
to
go
into
my
special
notes.
If
you
will
all
right,
I
hope
that
wasn't
confusing,
because
I
I
feel
like
I
stumbled
over
that
explanation
a
little
bit.
B
Right
so
back
to
our
cluster
here,
hey
everybody's
up
and
running:
let's
see
where's
my
node
extra
large
double
extra
large.
Here
we
go
yaml,
so
it's
got
our
paints
on
here.
Just
like
we
wanted
it
to.
We
can
see
it's
a
part
of
all
the
different
things
and
all
the
different
stuff
that
we
need
it
to
be
so
just
send.
As
an
example,
let's
go
ahead
and
move
our
router
I'm
just
going
to
copy
and
paste
these
commands.
A
B
Yep,
but
you
can't
do
it
all
through
the
GUI.
If
you're,
you
know
a
glutton
for
punishment,
I
guess
so
all
I've
done
here
just
instead
of
doing
OC,
git
Andres
controller
default,
you
can
see.
All
I
did
was
go
to
the
Ingress
controller,
custom
resource
definition
and
then
select
instances
and
the
only
instance
that
was
there
is
default
and
now
I'm
looking
at
the
ammo
for
it
so
just
like.
B
I've
been
switching
back
and
forth
between
Linux
and
mac
and
this
week
as
I
move
between
deaths
and
I'm
back
on
a
Mac
at
the
moment,
which
means
that
I
keep
pressing,
Ctrl,
C
and
Ctrl
V
instead
of
command
C,
because
you
know
Maria's,
real
and
painful
all
right,
let's
see
so
one
thing
somebody
asked
about
in
an
internal
Thread
about.
Is
there
a
Best
practice
or
a
suggestion
like
you
can
see
here,
there's
both
a
a
value
of
reserves.
B
Actually,
this
yeah
no
schedule
no
execute
with
a
value
of
reserved,
so
in
some
part
of
the
documentation,
creating
an
infrastructure
node
layout,
so
here
label
the
node
with
infrar
equals
and
then
an
empty
space
or
in
just
a
blank
string
and
then
with
the
taint
there's
a
couple
of
different
things
that
you
can
use
as
well
right.
So.
B
I'm
done
here
so
you
can
see
this
one
doesn't
have
a
value.
It's
just
an
empty
string:
OC
admin,
Adam,
taint,
node,
node,
name,
key
equals
value
and
effect
so
in
for
equals
reserved.
So
you
can
do
the
same
thing.
Infrared
equals
empty
string
here,
there's
no
real!
It
doesn't
matter
here.
You
can
use
infra
as
I
was
saying
before
with
the.
B
B
B
A
B
B
So
if
a
pod
is
already
running
on
the
Node,
when
you
apply
the
no
schedule
taint,
it
will
leave
that
pot
alone.
But
if
that
pod
is
terminated
and
then
goes
back
through
a
scheduling
process,
it
would
not
be
reassigned
to
that
node
without
having
a
taint
for
the
no
schedule
or
excuse
me,
a
toleration
for
the
no
schedule
no
execute,
on
the
other
hand,
will
evict
pods
from
a
node
that
don't
have
a
toleration
and
there's
a
you
can
configure
how
long
that
takes
based
on
a
number
of
things.
B
B
B
So
anyways
we'll
save
this.
Our
Ingress
controller
yep
reload,
okay,
we're
good
there!
So
let's
come
up
here
so
you
can
see.
The
GUI
has
helpfully
dropped
us
into
the
openshift
Ingress
operator
namespace.
If
I
come
up
here
to
workloads,
I,
don't
remember
if
the
Ingress
pods
are
also
in
this
namespace
I
think
they
are.
B
B
B
A
B
B
A
A
B
B
A
B
I
say
that
as
though
I
can
look
at
the
name
and
tell
exactly
which
one
it
is
here,
yeah
double
extra
large,
so
so
yeah
So.
Eventually
we
got
there
I
tested
this
yesterday
and
it
was
seamless
and
apparently
I
just
didn't
change,
something
or
forgot
to
change
something
when
I
copy
and
pasted
from
a
regular
machine
set,
anyways
I
tested
the
the
other
thing
as
well.
Control.
B
B
So
the
short
version
is
if
you're,
using
openshift
sdn
you're
using
vxlan
in
between
for
pod
to
pod
communication,
assuming
it
traverses
nodes,
I.
Think
if
it,
if
the
pi,
both
pods,
are
on
the
same
node
I,
don't
think
it
uses
the
sdn,
so
yeah
openshift
sdn
is
vxline
based.
So
if
it
traverses,
if
it's
pod
to
pod
traffic
between
two
separate
nodes,
then
it
will,
it
will
use
vxline
whether
or
not
you're
hitting
that
limit.
I
would
say:
you'd
have
to
test
that
yourself
again,
I,
don't
the
the
documentation
for
that.
B
So,
for
such
a
significant
thing,
the
documentation
is
very
light
on
like
conditions
where
it
happens
outside
of
vxlan
and,
like
anecdotally,
I
have
seen
where
people
are
able
to
get
more
than
that.
So
I
feel
like
this
is
definitely
something
that
you
would
want
to
test
and
and
verify
for
your
application
and
your
you
know
your
servers,
your
cluster
deployment
as
to
whether
or
not
that's
being
affected.
B
So
I
I'm
I'm
curious
as
to
whether
or
not
you
know
if
this
is
true,
you
know,
were
we
seeing
it
before
and
didn't
realize
it.
You
know
if
I'm
using
you
know
if
I
was
using
other
vxlan
based
Technologies
I,
know,
there's
multiple
different
Hardware
vendors
that
support
vxlan
offload
stuff,
like
that.
Maybe
it
was
being
worked
around
that
way.
I
don't
know
so,
like
I
said,
I
just
feel
like
there's.
B
There
needs
to
be
more
details
in
the
documentation
other
than
you
know,
yeah
it's
if
you're
using
the
xlan
one
gigabit
limitation
like
I,
promise,
yeah
I'm
gonna
go
with
the
trust
but
verify
okay,
so
yeah
that
that's
kind
of
all
there
is
to
it
as
far
as
moving
workloads.
It's
just
like
openshift
infoser
services
are
just
like
any
other
application
workload.
You
know
you
affect
scheduling
via
taints,
tolerations,
node
selectors,
you
know
the
same
Affinity
anti-affinity.
B
Yeah
I'm
I,
don't
know
if
anybody's
noticed,
I
feel
like
I'm
I'm
I'm,
trying
to
do
a
little
bit
more
of
like
the
walking
through
the
process
and
not
the
cooking
show
side
of
things,
because
you
know
for
a
long
time.
In
the
streams
it
was
oh
yeah,
here's
this
really
cool
thing
and
I
figured
all
this
out
and
copy
and
paste
everything
in
and
I'm
trying
to
be
a
little
more
in
depth
with
it.
So
I
don't
know
if
anybody
has
opinions,
thoughts
on
it.
B
Let
us
know
again:
you
can
reach
out
chat
for
the
next
few
minutes,
while
we're
still
here,
but
you
can
also
ping
us
on
email,
various
social
medias.
Let
me
find
the
right
button
here
to
press
andrew.sullivan,
redhat.com
or
Johnny
j,
o
n
n
y
at
Red
hat.com,
but
yeah
with
all
that
being
said,
I
don't
have
anything
else.
What
we've
got
coming
up
in
the
next
few
weeks,
Ronnie
we've
been
talking
with
a
number
of
people
about
a
number
of
different
things
happening
on
the
show.
So.
A
Yeah
I
mean
so:
we've
got
a
few
weeks
from
now.
We've
got
for
sure.
Oh,
my
god.
Oh
we
have
Rosemary
from
hash,
Court
bolts,
just
gonna
come
on
and
she's
going
to
talk
to
us
about.
You
know
essentially
give
us
a
run
through
of
like
what
hash
Court
vault
is
in
kubernetes
and
openshift,
and
then
how
we
can
use
it
in
openshift
and
some
of
the
benefits
so
I
think
that's
gonna,
be
a
really
cool
demo.
She's.
A
A
Right,
that's,
you
know,
that's,
that's
all
I
I
can
ask
for
you
know,
but
then
what's
the
other
one,
we
have
a
couple
partners
that
we've
been
talking
to
that
we're
we're
trying
to
like
confirm
some
stuff
but
yeah.
That's
that's
the
big
things.
We've
got
a
few
partners
and
a
bunch
of
good
stuff
coming.
It's
just
a
matter
of
getting
schedules
locked
down.
B
A
B
B
Operator's
reference
that
was
easy,
so
I'll
paste
this
into
the
chat
here.
You
know
using
the
search
for
what
it's
meant
for,
but
you
can
see
here
in
the
table
of
contents.
It
goes
through
for
all
of
those
cluster
operators
so
from
the
CLI.
If
you
do
an
OC
get
Co
or,
as
you
saw
from
my
console
here,
if
I
come
here
to
cluster
settings
cluster
operators.
B
A
And
I'm
going
to
share
a
link,
also
just
in
case
you're,
talking
about
building
operators
because
I'm
not
I'm,
not
100
sure,
but
if
you're
talking
about
building
them,
here's
the
operator
SDK,
which
has
the
different
varieties.
So
you
can
either
do
it
through
Helm
or
ansible
or
go.
This
is
the
go
Edition,
but
if
you
just
dig
through
the
documentation,
they
walk
through
actually
building
an
operator
and
all
the
resources.
So
it
just
depending
on
which
way.
You're
asking
that
question.
A
Andrew,
just
real
quick,
we
had
a
question
for
Mira
and
I'm.
Sorry,
if
I
butchered
your
name,
a
training,
certification
question
from
4.6
exam
to
it
is
the
4.6
exam
two
different
from
4.10.
Were
there
many
features
added
to
4.10,
I
I
would
say
personally
an
Andrew
stump
on
this
right,
like
if
you're
gonna,
if
you
can
take
the
exam
on
4.10
I,
would
take
that
because
there's
some
API
changes
that
happen
in
4.9
that
won't
be
accounted
for.
A
B
So
what
this
shows
us?
Hopefully
it
stays
readable
as
I
expand
this
to
make
it
readable.
So
what
this
shows
us
is
for
each
of
the
cluster
operators
here,
and
you
can
see
this
big
old
list
that
goes
down
through
here
or
for
each
of
the
images.
I
have
a
GitHub
repo,
that's
associated
with
it.
So
let's
pick
on.
B
B
B
But
these
are
interesting.
These
GitHub
repos
are
interesting
because
a
lot
of
times
the
engineering
folks
who
work
on
these
will
include
a
lot
of
a
different
information
inside
of
here.
So
I
would
say,
sometimes
be
careful
because
there
might
be
unsupported
things
that
are
in
here
as
they
go
through
and
document
like
how
to
do
development
on
it
and
all
this
other
stuff.
B
B
One
thing
to
check
for
all
of
these
that
I've
looked
at
all,
have
branches
for
each
release,
so
we're
on
4.12
right
now,
so
you
would
go
to
the
4.12
release
and
this
would
be
you
know
the
branch
that
it
uses
for
for
our
for
ours,
because
it
does
change
especially
MCO.
You
know
the
different
files
that
they
manage
automatically
through
MCO
changes
over
time
and
so
pay
attention
to
which
release
you're
actually
looking
at
there.
B
B
Yep,
okay,
so
I
think
we
got
to
all
the
questions
yeah.
What's
new
in
4.10,
thank
you
for
the
link
to
that
or
hope
nine
you
would
have.
You
would
also
want
to
look
at
the
what's
news
for
all
of
the
versions
in
between
there
as
well.
So
four,
seven,
eight
and
nine,
because
we
don't
have
cumulative
versions
of
those
I,
think
that's
the
right
word.
B
Would
there
be
an
episode
on
how
to
add
custom
catalog
operators
to
Cluster
add
them
in
disconnected
clusters.
We
touched
on
that
during
the
disconnected
Deep
dive,
which
is
way
back
in
the
40s
I.
Think
I
think
we
ran
out
of
time
when
we
did
that,
though
so
I
don't
think
we
got
in
depth
with
it.
It's
also
changed
quite
a
bit
now
that
we
have
the
OC
mirror
command
and
stuff
like
that.
So
it's
a
good
call
out.
B
All
right
so
I
I
got
I
got
nothing
again.
If
you,
if
anybody
is
you
know,
whoever's
watching
us
live
if
you're,
not
watching
us
live
anything
comes
to
mind
so
feel
free
to
reach
out
to
us,
Johnny
and
I
love.
To
hear
from
you
all
and
also
I'll.
Add
in
the
kubernetes.
Slack
is
a
great
way
to
reach
out
to
us
we're
both
on
there
and
I.
Try
to
keep
up
with.
What's
going
there,
yeah
I
know
it's
it's
the
it's
a
community
side.
B
So
there's
a
lot
of
okd
questions.
That
okayd
is
just
different
enough
for
me
that
I
I,
I
feel
unconfident
or
not
I.
Don't
feel
confident,
answering
questions
all
the
time,
but
but
yeah
other
than
that.
We
will
be
back
next
week.
Yeah,
there's
nothing
we'll
be
here
next
week,
yeah
I
gotta,
think
about
it,
because
I
feel
like
between
one
or
both
of
us
there's
been
so
much
going
on
that
we've
missed
missed
here
and
there
it's.