►
Description
OKD4 on Azure
Josef Meier ( Rohde and Schwarz)
OKD WG Briefing
March 2020
Link to Slides: https://github.com/openshift-cs/okd.io/blob/master/briefings/slides/okd-4.4-on-azure_JosefMeier.pdf
A
Hello
and
welcome
to
this
okd
working
group
briefing
on
deploying
okay
d4
on
Azure
brought
to
you
by
Joseph
Meyer
who's
been
working
with
the
working
group
testing
on
one
of
the
many
platforms
and
he's
gonna
walk
us
through
today.
His
lessons
learned
deploying
the
okay
d4
beta
release
now
on
Azure,
so
Joseph,
why
don't
you
introduce
yourself
and
take
us
on
a
tour.
B
B
My
company
is
Raza
old,
it's
founded
in
1933
and
it's
famous,
all
almost
every
physician
or
electronics
engineer
should
know
it,
because
we
are
one
of
the
leaders
in
making
measurement
equipment.
Oscilloscopes
signal,
analyzers
spectrum,
analyzer
signal
generators.
We
also
make
TV
transmitters
where
we
are
market
leader.
It
could
be
that
also
in
in
Canada
you
a
TV,
comes
from
our
transmitters.
We
also
make
body
scanners.
B
Maybe
you
have
seen
one
on
an
airport.
We
also
do
lots
of
RF
high
frequency
measurement
and
generation
equipment.
Thats
sets
what
we
do
and
my
job
since
two
years,
almost
three
years,
I'm
working
in
a
team
which
is
yeah
working
with
transformation,
a
draw
and
Schwartz
and
our
job
is
to
build
how
it,
how
we
call
it
a
rohde,
&
schwarz
cloud,
which
is
based
also
on
okay,
D
or
open
shift.
B
Why
do
we
wait
eagerly
for
that,
because
yeah
we
have
more
and
more
applications
which
require
more
modern
kubernetes
version.
As
you
know,
in
version
3.1
is
the
last
version
in
the
the
version
3
yeah
line
of
software
of
okd,
its
kubernetes
1.11
and
yes,
ok,
t4
is
starting
with
1.17
as
far
as
I
know,
and
that's
what
a
lot
of
applications
require.
Now
that
we
have
modern
kinetise
version,
what
we
also
like
Ozzie
great
ops,
features
of
ok
t4
will
see
automatic
updates,
not
only
for
okd
but
also
say
underlying
OS.
B
B
Everything
looks
they
are
like
it
wasn't
designed
for
kubernetes.
It's
I
think
the
best
web
UI
nowadays
for
kubernetes
in
the
world
and
also
Celia
wrote
my
business
are
very
promising.
The
last
time
I
saw
a
powerpoint
presentation,
with
more
than
100
slides
filled
with
nice
cool
features
which
are
coming
the
next
releases
and
yeah.
That's
why
we
like
to
jump
onto
okkadine
for
once.
That's
also
the
reason
why
I'm
very
engaged
in
getting
that
because
I
like
to
see
it
in
my
company.
B
We
have
two
versions
were
locations
where
the
use
of
mushaf
okay
d,
as
this
is
a
truly
inch
watts
on
premises
in
munich
and
also
on
asia,
and
we
do
some
kind
of
hybrid
approach.
It
means
we
build
software
on
premises
and
deploy
it
hanji
in
the
public
cloud.
That's
why
I'm
working
on
the
vSphere
version
of
Acadia
and
also
on
the
azure
version,
the
azure
version
is
kind
of
special
because
yeah
it
was
was
hard
to
get
it
running
because
yeah
there
were
several
yeah
milestones
which
had
to
get
achieved.
B
First,
one
was
the
first
problem
was
had.
There
was
no
Fedora
chorus
version
which
worked
out
of
Suboxone,
Asia
series
and
I
have
to
say
there
was
a
network
pack
which
required
a
reboot,
sometimes
even
more
on
the
first
boot
before
the
virtual
machine
got
an
internet
connection.
I
think
not
only
Asia
is
affected
by
that,
but
yeah
it
was
my
problem
and
I
at
first
I
yeah,
he
booted
every
VM
I
had
no
idea.
If
he
open
shift
control,
because
I
kubernetes
control
plan
was
still
running,
it
was
not.
B
It
was
not
ill,
but
since
a
few
days
there
is
a
test
version
of
Fedora
core
is
available,
whereas
this
box
seems
to
be
fixed,
I'd
routed
a
tried
out
several
times
to
deploy
a
VMs
with
Fedora
chorus,
and
it
worked
every
time.
So
this
problem
seems
to
be
fixed,
I'm
very
interested
to
get
it
in
our
released
version
soon.
The
second
point
Ezreal
said:
Zadora
chorus.
B
In
my
case
it
was
yeah.
I
have
my
internet
connection
here
at
home.
There
I
do
the
most
work
on
ok,
D
is
very
slow,
I
have
download
rates
of
five
Giga
Viking
about
nice,
5
megabyte
per
second
upload
rate
is
far
less
and
it
means
if
I
use
the
Installer
without
modifications.
Then
it's
timed
out
very
almost
always
after
half
an.
A
B
So
I
wasn't
able
to
use
it
and
what
it
made
but
made
it
also
complex,
is
said,
see
if
it
were
a
coarse
image.
A
few
weeks
ago
had
to
be
modified
server.
Nests
modifications,
necessary
risk,
Wemo,
but
also
sis
has
fixed.
A
christian
told
me
to
death
said
it
should
be
fixed
since
a
few
weeks.
I
was
surprised
about
that
because
yeah
I
made
a
work
around.
B
There's
lots
of
love
in
the
long
weekend
and
to
get
around
that,
but
I
think
it's
it's
still
necessary,
because
I
tried
today
to
keep
cloak
ID
on
Asia
without
modifications
we
see
installer
from
github
and
it
didn't
work,
maybe
I
can
can
show
Christian
later
what
what
the
problem
is.
So,
yes,
I
made
a
workaround
for
the
all
those
problems
here,
because
I
wanted
to
test
okay,
Deion,
Asia
and
I.
Don't
want
had
lots
of
to
do
lots
of
manual
interaction.
B
All
what
I
want
to
show
you
in
the
next
three
slides
is
not
necessary
anymore.
If
it
water
core
is,
is
available
on
the
art,
Asia
marketplace,
that's
important
to
say
this
yeah
modification
from
mine
will
never
get
it
to
master
into
installer.
It's
just.
If
you
want
to
try
our
to
ok
donation,
yeah.
Here's
a
little
picture
should
explain
what
situation
we
have.
If
Fedora
Co
s
would
be
on
the
edge
of
marketplace,
you
could
create
okay
D
only
by
colleagues
as
command.
B
Openshift
install
create
cluster
because
we
don't
have
it
now
on
the
marketplace.
I
made
a
yes
Vadim,
called
it
a
hacky
workaround.
It
is
about
Hecky.
Workaround
I
be
honest
with
that,
but
it
helped
me
helped
me
a
lot.
What
does
this
work
around
you?
It
creates
a
helper
VM
on
Asia
through
the
installer
and
such
help.
Early
a.m.
B
downloads
see
if
it
or
a
chorus,
PhD
file,
extracts
it
and
uploads
it
to
the
storage
account
in
Asia
I
use
terraform,
which
is
included
in
the
open
shift
installer
to
do
that,
and
my
assumption
was
sets
it
download
and
upload
is
much
faster
if
I
do
it
in
a
virtual
machine
which
is
already
on
Asia,
and
that
is
also
the
case
for
that
I
had
to
modify
the
Installer
yes,
yeah
well.
I
also
thought
about
about
different
strategies
to
do
to
achieve
the
goal.
B
I
also
saw
that
I
could
manually
upload
the
extracted
image
to
Azure
blob
storage
and
yeah,
but
the
problem
is
still
the
other.
The
upload
is
very
slow
in
my
location
here,
so
yeah
I
I
modified
the
Installer
and
said
sir.
What
I
want
to
show
in
the
next
three
slides
the
core
commands
which
are
necessary
to
run
in
this
help
of
the
M?
The
list
is
very
short,
because
I
removed
all
the
queima
stuff
from
that
in
principles.
The
script
runs
on
the
helper
BM.
B
I
download
are
a
set
copy,
it's
a
tool
from
Microsoft
for
Asia,
where
you
can
upload
software
to
Z
as
a
container
blob
storage.
I
download
CF
course
the
compressed
F
course
VHD
file
I
extracted
and
upload
it
this
as
a
copy
to
a
Phillip,
blob,
storage,
yeah
and
on
asia.
I
get
wonderful
upload
rates.
It
only
takes
minute
and
I
think
I'm
Jewesses
tasks
here
at
home
I
never
got
it
running.
It
runs
forever.
B
The
next
don't
be
frightened
by
the
list.
Here
it's
also
nothing
very
special.
You
have
to
clone
the
overshift
installer
at
checkout,
CF
course
branch.
Then,
yes,
that
no
it's
it's
still
series
a
wrong
release.
Image
addressed
in
the
Installer
I
think
what
he
made
a
commit
for
that,
but
it
hasn't
landed
so
I
have
to
replace
release
4.5
by
4.4
I.
Think
maybe
two
more
roses
is
not
necessary
anymore.
Then
I
cherry
pick
3
commits
from
my
repository
where
I've
made
he
changed
for
the
Installer.
B
B
That's
everything
about
the
heck,
I
think
yeah.
This
here
should
be
very
common
at
first,
you
have
to
create
a
service
principle
on
Asia.
There
is
a
very
good
documentation
about
that
in
the
OpenShift
documentation.
I
have
pasted
the
link
to
the
slides,
and
this
is
very
important
as
service
principle
is
something
like
a
a
machine
service
account
which
has
some
Arabic
on
it,
which
allow
a
selfish
eft
installer
to
create
resources
on
Asia.
You
have
to
do
this
first
and
you
get
also
an
application
ID,
a
secret.
B
You
need
your
attendant
ID
and
your
a
subscription
ID.
This
is
our
measure
terms.
Everything
used
to
Azure
should
know
them,
and
you
will
need
them
later.
A
next
step
is
that
you
have
to
generate
SSH
key
pair.
This
HR
key
again
afterwards,
I
create
an
open
shift,
was
over
shift
installed,
create
install,
configure,
create
a
config
file
which
is
called
install,
config
dot.
B
Why
I
don't
know
it
does
really
that
that's
why
I
save
it
away
and
copy
it
back
later.
If
I
do
the
next
test,
what's
also
important,
the
what
we
install
the
cluster
is
that
you
have
to
set
an
environment
variable
which
overrides
a
default
Fedora
core
SOS
image
with
one
the
Fedora
core,
as
team
has
offered
a
few
days
ago,
which
contains
the
azure
Network
yeah
fix.
B
B
With
all
these
preparations
yeah,
you
can
create
a
cluster.
This
should
be
very
yeah.
You
should
know
this
command.
It
takes
around
10
minutes
at
first
sorry
at
first,
so
installer
will
create
this
VHD
help.
Ibm
I
am
I,
had
told
about
told
you
about
that.
Before
and
in
a
few
minutes
later,
yeah
we
will
be
the
HD
file
was
8
gigabyte
will
be
in
a
storage
account
in
Asia
and
after
that,
the
Installer
proceeds
as
usual.
B
It
creates
an
image
on
Asia,
Bhd
and
Asia
its
re,
an
edge
ovm
image
out
of
this
VHD
file
and
all
the
VMS
are
created
and
after
around
10
minutes,
say
bootstrap
PM
should
start
after
the
master
the
MS
show
up
in
the
azure
portal.
You
can
start
watching
your
ok
d
for
a
cluster
coming
to
life
by
using
OC
CLI,
and
I
think
it's
very
nice
that
the
installer
creates
all
the
resources
completely
automatic.
So
you
don't
have
to
do
anything
manually
normally
if
everything
works
yeah
and
after
think
of
an
hour.
B
That's
very
important.
This
is
why
I
broach
this
modification
to
install
of
4.
It
should
look
like
this.
In
the
end,
there
is
a
an
F
course
VHD
file
and
now
Asia
blob
storage,
container
I
always
had
to
use
the
blob
type
page
plop.
Without
that
and
Asia
complains
that
for
images
it
needs
page
block,
not
block
blob
I
saw
in
the
sources
that
E
and
the
F
course
branch
and
also
see
master
branch,
have
block
blob
in
the
source
code.
B
B
So
if
you
want
to
watch
that
installation,
you
have
to
get
COC
CLI
tool
for
version
4
of
okd,
it's
very
important.
Don't
use
your
old
OSC
command.
If
you
only
want
to
see
ports
or
C
as
ours
visit,
all
the
old
version
would
work.
But
if
you
want
to
get
upgrades
for
your
cluster
for
your
running
cluster
C
version,
3
CLI
Co
see
it
wouldn't
work
so
I
get
it
here
at
4k
d,
4i
antar
it
and
then
I
set
an
environment.
B
B
Oh
si
can
get
a
connection
to
the
cluster
because
jib
config
are
also
certificates,
and
here
the
cluster
location
to
my
name
is
also
included.
What
I
normally
do
afterwards
is
to
do
a
watch,
so
I
see
all
pots
in
all
namespaces
each
second.
That
helps
me
seeing
any
problems.
If
pots
don't
come
up
where
we
start
and
normally
I
am
after
what
I
am?
If
I
have
the
situation?
I
look
in
the
locks
reports
and
if,
if
it
looks
a
strange,
can
I
open
a
ticket,
the
last
you
yeah
days
weeks
said
didn't
happen.
B
Everything
was
running
as
expected.
Yes,
but
you
can.
You
should
also
have
a
look
on
that,
if
see
pots
in
the
name,
space,
OpenShift
machine
API
already
normally
he
shoots,
he
created
three
worker
VMs
in
Asia
and
after
a
while
say
the
CSRs
for
them
are
generated
and
a
few
seconds
later
normally
say:
I'll
join
the
cluster
and
if
set
happens,
you
are
almost
done
with
everything
problem.
Is
it's
a
little
spoiler
that
since
since
yeah
that
sometimes
the
VMS
come
up
in
asia,
but
they
don't
do
anything?
B
Yeah
happened
to
me
today,
two
times
that
server
to
worker
VMS
joining
the
clusters,
a
third
one
didn't
join.
The
cluster
I
was
able
to
SSH
into
them.
I
saw
that
they
have
an
internet
connection,
but
soorapadman
images
didn't
show
any
images
and
it
yeah
also
the
journal.
The
system
since
ronald
CTL
didn't
show
anything
any
didn't
show
any
great.
B
It
was
idling
around
I,
don't
understand
where
it
comes
from.
Maybe
it's
a
new
effect.
We
have
to
see
it's
a
live
game
where,
if
it
still
happens
again
and
afterwards
after
around
half
an
hour,
you
should
be
able
to
see
the
pep
Qi.
Here
is
a
screenshot
from
a
cluster
I
made
two
days
ago
and
also
yeah
I
can
show
that
maybe
I
can
shows
it
before
we
go
into
a
live
action.
I
will
switch
the
presentation
to
my
browser.
B
B
Here
it
is
okay
here
something
has
degraded
already
I.
Think
it's
a
simple
operator,
and
here
we
are
on
Asia
yeah
I
can
see
it
inside
us.
No,
it's
at
our
our
domain
name
and
if
I
go
into
the
settings,
I
see
said
I'm
on
this
version
it's
from
two
days
ago,
I
think
I
think
it's
maybe
it's
the
beta
version,
I'm
not
sure.
B
Here
we
have
a
D
created
samples
operator
that
happens
from
time
to
time.
After
an
update
that
something
is
degraded,
I,
don't
know
exactly
what
I
can
do
against
it.
If
it
is
normal
because
upgrades
as
far
as
I
know
are
not
supported
at
the
moment,
and
because
the
tests
for
upgrades
have
recently
begun
a
few
years
ago
can
see
it
on
the
release
page
for
400
kg
for
origin.
I
love
this
visualization
here
and
yeah.
That's
I!
Think
that's
is
it's
the
reason
why
upgrades
sometimes
are
working,
sometimes
not
but
yeah?
B
Normally,
it's
the
first
installation
always
works,
and
everything
is
green
though
said
this
should
be
her
proof
that
okay,
I
can
run
I,
also
deployed
our
go
CD
as
a
test.
Everything
is
running
aqua
CD
here
is
it
we
use
a
go
CD
and
get
ups
for
configuration
of
the
cluster
and
also
for
configuration
and
automatic
deployments
of
our
applications.
Maybe
maybe
I
will
talk
about
that,
because
we
have
had
a
few
nice
finding
about
it.
B
B
B
I,
don't
want
to
show
how
I
built
the
Installer
the
verse
instructions
in
the
presentation
work.
I
have
tried
it
several
times.
You
can
try
it
on
your
own.
If
you
like,
I
will
go
in
a
folder
here.
Is
we
see
two
already
downloaded
and
now
yeah
I
know
I
already
compiled
the
OpenShift
installer,
with
my
modifications,
I
check,
if
it
is
not
here,
I
will
get.
My
environment
variable
come
on
my
environment
for
the
other,
for
the
Fedora
chorus
image.
If
you
don't,
if
you
forget
it,
you
will
have
problems
later.
B
B
B
I
have
to
answer
a
few
questions
here.
I
am
asked
for
a
public
key
though
it's
it's
useful
for
debugging.
If
you
have
a
problem,
you
can
assist
agent
to
EMS
which
make
trouble
I
always
create
them
here.
A
selected
area,
I
already
was
locked
in
before
I
entered
my
credentials,
I
I'm,
pretty
sure
you,
you
know
what
to
enter
here.
You
have
to
enter
the
service
principle
information
about
the
service
principle
you
created
with
help
of
CEO
of
shift
documentation.
B
B
A
B
And
the
credentials
are
stored
in
the
home
directory
in
a
hidden
directory
called
Asia,
see.
How
are
the
credentials
I
say
are
not
in
the
Installer
Yama
now
I
can
create
a
cluster
hope.
I
did
not
forget
anything,
always
brutal.
If
you
see
that
you
forget
something
in
the
very
start,
if
you
already
have
all
the
M
spawned,
that's
very
sad,
always
I
hope,
I
didn't
forget
anything
now,
as
the
infrastructure
gets
created
by
the
Installer
in
Asia.
B
What
you
see
here
is
I
forgot
to
switch
off
my
debug
code,
yeah
I
have,
as
I
told
I
am
fast
as
heck,
which
creates
a
helper
VM
and
now
terraform,
which
is
a
part
of
the
Installer,
creates
few
resources
and
also
starts
with
sis
help.
Vm
go
to
Asia
cam,
wait
until
you're,
some
c4,
ok,
d4
resource
group
will
pop
up.
It
always
takes
a
while.
B
So
it
doesn't
not
mean
that
as
the
Installer
is
doing
nothing
to
Asia,
you
I
always
takes
a
few
seconds
or
sometimes
minutes
until
it
shows
resources
which
were
already
created.
So
we
can
see,
sets
here
or
some
Ora
tries
of
create
opt
for
cluster.
Let's
remember
this
to
Andrey
and
a
few
seconds
as
they
should
pop
up
a
new
entry,
so
maybe
I
can
go
first,
yeah
what
it
normally
do.
If
things
go
wrong,
it's
this
stage
it
happens
or
a
sure
deployment
from
time
to
time.
B
Yes,
these
are
the
typical
commands,
which
will
sometimes
save
your
life.
You
should
you
could
enter
at
first.
If
he
mastered
EMS,
don't
come
come
up
who
are
still
for
a
long
time
in
creating
date,
then
you
should
enter
see
bootstrap
p.m.
see.
Username
is
always
core
the
public.
He
is
the
one
you
have
created
or
with
ssh-keygen.
B
Always
next
thing
I
do
is
set
at
were
suitable
portman
images
to
check
if
see
if
it
work
or
is
download
something
from
the
internet
after
there
are
four
or
five
images
downloaded
normally
as
it
would
sleep
well,
every
every
M,
a
restart,
I.
Think
it's
because
and
say
are
some
ins.
Images
are
some
tools
which
are
downloaded
and
written
to
to
the
Fedora
core
as
partition
and
after
reboot
sees
tools
are
permanently
inside
of
see.
Pedro
SVM
also
cry
control
is
not
in
CDM
from
the
beginning.
It
needs
a
reboot.
B
And
after
that,
it's
it's
in
the
system
with
Cisco
Sulochana
control,
no
pager
follow,
you
can
see
all
locks
all
system
locks,
and
here,
if
you
have
containers
which
make
problems,
you
can
have
a
look
in
the
in
the
locks.
I
forgot
the
command,
which
shows
you
running
containers,
it's
cry:
control,
PS,.
B
A
they
can
see
the
container
it
is
if
it
takes
forever
until
the
control
plane
comes
up,
saying
you
have
a
few
options.
The
bootstrap
VM
has
already
a
public
IP.
The
master
DMS
I
are
behind
a
load
balancer.
So
if
you
want
to
go
into
a
master,
VM
or
lock
a
VM,
you
have
to
give
Sam
a
public
IP
and
also
enable
port
22
in
the
network
security
group.
B
That
also
a
point
where
I
think
it's
better
debugging
could
be
improved
if
it's
a
private
key
for
debugging
purposes
would
be
in
the
bootstrap
iam,
because
all
the
MS
are
in
the
same
private
network.
It
would
be
nice
if
I
could
jump
to
each
VM
masters
and
workers
from
the
bootstrap
iam,
but
currently
that's
not
possible.
Maybe
it
already
changed.
I
have
not
tried
it
since
the
last
days,
but
normally
private
key
is
not
inside
of
supporter
p.m.
which
make
sense
for
security
reasons,
but
for
debugging.
It's
so
nice,
though.
B
If
the
next
step
we
can
see
with
OC
get
csr
if
sell
some.
If
I,
all
masters
are
already
approved-
and
this
is
yeah-
see
situation
where
control
flame
should
should
be
there,
and
you
could
should
see
something
without
see
get
get
pots.
You
should
see
that
a
lots
of
containers
are
creating
if
Sara
after
reasonable
time
only
are
not
three
masters
than
you
have
are
lost.
B
B
B
Ok,
we
have
only
one
virtual
machine.
Let's,
let's
have
a
look.
Ok,
it's
still
running
if
one
will
actual
machine
as
this
is
this
helper
VM
I
always
talking
about,
and
we
have
a
storage
account
ah-yi
EMM,
which
is
already
created.
It
means
at
this
point
I
could
delete
the
help
of
e/m.
It's
not
automatically
deleted.
I
saved
this
work.
Paulo!
Don't
want
to
see
that
to
show
you
account.
B
B
So,
let's
refresh
yeah
yeah,
we
have
yeah,
you
have
bootstrap
vm
and
three
masters
at
this
moment.
What
I
normally
do
in
this
phase
is
to
go
to
the
bootstrap
iam.
It's
still
creating
and
I
love,
Caesarea
console
where
you
can
see
if
it.
If
it's
running
what
VM
does
currently
it's
still
creating
yes,
complaints.
B
B
B
Being
creation
always
takes
a
little
bit,
what
is
important,
you
should
not
use
this
URL
to
to
SSH
into
it
cause.
This
is
the
IP
address
of
the
load.
Balancer
go
into
networking
and
take
this
IP
addresses
IP
address
of
the
virtual
machine.
We
also
see
that
the
bootstrap
PM
already
has
what
22c
SSH
port
opened.
B
B
B
B
Already
inside
but
yeah
what
I
saw
I
don't
know
if
it
is
severe
or
not.
Normally
it
isn't
that
I
have
sometimes
faith
units
here
of
services
which
didn't
came
up,
yeah
I
talk
too
much.
This
is
the
situation
here
where
supports
3
p.m.
restarts.
I
said
it
before
it
installs
a
few
tools
from
the
images
it
downloaded
from
the
internet.
It
extracts
the
tools,
can
put
it
to
see
a
12
second
partition
I
think
it's
a
way.
Our
pmos
Westry
works
reboot,
and
then
you
have
such
words
inside
of
CBM.
B
B
A
favorite
command
here,
good
cause,
say
API
server
already
is
answering
as
far
as
I
understood,
the
bootstrap
VM
starts
a
fake,
a
fake
control
plane
and
all
the
masters
get
there
ignition
convicts
from
the
boots
heavy
em.
They
get
yes,
a
install
themselves
with
software
reboot
and
joins
a
cluster,
and
if
there
are
three
master
VMs,
if
they
have
joined
the
cluster,
then
they
push
through
p.m.
stops,
stops
working
and
gets
deleted
and
that's
a
point.
B
B
B
C
B
Period
said
this
can
last
a
little
bit
as
I
not
see
any
CSRs.
This
no
worries
about
this
errors,
which
are
warnings.
The
terraform
feature
it
tells
me
said
there
are
some
resources
are
deprecated.
There
is
a
already
aversion
to
of
the
agile
provider
for
terraform.
He
already
tried
it
out,
looks,
looks
very
promising,
and
this
information
tells
me
that
I
think
it's
deprecated,
it's
not
a
not
on
regal
error.
B
Our
CEO
sauce
come
to
life.
It
is
always
for
me.
It's
I'm,
not
I,
don't
know
the
insides
of
okay
a
very
good,
but
this
is
for
me
the
first
indications
that
something
tries
to
attend
the
cluster,
the
good
signal,
because
it
means
that
all
master
VMs
yeah
came
up,
which
would
also
be
visible
now.
Oc
get
notes
that
it
is
true,
wait
yeah,
they
are
not
ready.
It's
it's.
Okay,
I
hope,
I,
don't
get
you
sick
through
my
switching
here.
B
B
It
was
wondering
on
the
last
try.
I
did
not
see
any
overshift
on
Zoe
operator,
don't
know
where
it
where
it
is
if
it
comes
later
or
because
the
last
try
an
offset
workers
didn't
came
up,
I
I
was
thinking
of
that.
This
is
a
problem.
Maybe
zetsubou
shift
console
operator
is
not
present,
but
I'm,
not
sure
yeah
how
its
exploding
you
see
that
isn't
that
cool?
B
B
They
won't
be
worried
if,
during
the
installation,
operators
get
decorated,
normally
say
repair
themselves,
sometimes
I
think
there
are
dependencies
between
operators.
So
yeah
don't
get
confused
if
serious,
that
you
created
status,
set
or
true.
What
concerns
me
more
is
set
series
they'll
know
all
operator
I,
don't
know
what
I
missed
here
yeah.
Maybe
it
comes
so
now.
I
will
check
if.
B
B
Xr
was
the
pot,
was
so
young,
it's
a
little
bit
if,
if
the
OSI
class
CLI
blocks
or
does
what
it
yeah
or
CNC,
was
this
message
don't
be
concerned?
It
can't
be
that
serious,
a
situation
where
the
load
balancer
kicks
the
bootstrap
p.m.
out
of
the
cluster
and
it
sometimes
fizzes
a
result
of
that
normally
notes
should
be
ready,
yet
I
think
it.
This
was
at
the
time
was
a
bootstrap
iam
left
us
sometimes
also
see
API.
What's
restart,
then
you
get
strange
messages,
but
don't
be
concerned
about
that.
B
It's
at
least
at
least
in
my
experience.
It's
normal
okay
check
again.
If
the
workers
come
up
here,
they
are,
there
were
now.
There
is
a
little
bit
tension,
because
it's
not
here
if
they
all
get
in
running
state
and
if
say,
are
in
running
state,
it's
not
clear
if
they
will
join
the
cluster
ever
I.
Ask
you,
as
I
said
a
few
days
ago.
It
worked,
but
but
I'm
not
sure.
If
I
had
had
luck
or
not
wait
a
little
bit.
B
B
Only
if
ignition
has
completed
its
configuration.
That's
a
point
where
CPM
goes
into
running
state
if
ignition
waits
for
its
configuration
file,
because
autos
if
workers
need
a
configuration
file
which
is
served
at
this
time
from
the
control
plane.
The
Masters
as
through
a
service
which,
which
name
is
a
machine,
config
demon,
I,
think
it
and
yeah
as
soon
as
say,
gets
this
configuration
say,
will
configure
themselves
and
say
our
intronic
state
may
be
therefore
chair.
B
No,
we
have
no
chance
to
see
here
something
in
the
serial
console,
because
error
needs
boot,
Diagnostics
to
be
enabled
maybe's
what
he
permits
also
a
good
idea.
If
I
could
configure
that
that,
because
masters
the
masters,
the
bootstrap
node,
it's
enabled
on
the
workers,
it's
not
enabled
for
some
reason
so
I.
Never
it
only
to
be
able
to
soak
my
god.
B
A
B
A
B
B
A
B
B
First
one
yeah
a
few
words
about
that.
If
you
do,
if
you
are
watching
much
cuts
are
running,
we
will
see
said
there
are
lots
of
pending
pots
yeah
it's
sometimes
there
are
a
lot
of
painting
pots,
because
some
pots
have
note
constrains
note.
Selectors
on
them
set
say
only
allowed
are
allowed
to
run
on
certain
note
and
sets
a
point
when
they
get
started.
Yeah,
if
say,
workers
come
come
up,
I
still
see
no.
B
B
B
B
B
C
C
B
B
B
B
As
some
of
this
should
do
the
job
as
something
this
warning,
if
you
upgrade
it
a
running
cluster
with
this
command
here,
I
think
it's
also
in
the
documentation.
Then
it
can
be
currently
in
the
beta
and
that
you
get
a
decorated
operators
but
I
think
yeah,
as
I
explained
before
it's
in
this
phase
of
of
the
project.
Where
upgrades
are
yeah
where
they
began,
the
team
began
to
start
a
test,
upgrade
it's
normal
settings,
sometimes
get
degraded.
And
yes,
it's.
It
was
the
installation
process.
B
B
The
rich
class
managed
premium
it's
edge
of
disk,
the
default
storage,
lots
of
kubernetes
does
the
wrist
twisted
oceans,
and
I
think
I
have
some
storage
test.
Pvc.
Here's
my
PVC
test,
I
created
a
GBC
assistant
volume
claim
and
it
is
found
it
created
a
persistent
volume.
There's
something
special
I,
don't
understand
it
with
this.
I
wanted
feature
that
if
you
create
a
PVC
st.,
it
will
only
bind
to
a
persistent
volume
if
it
is
used.
So
I
had
to
create
finalize
a
test.
B
I
had
to
create
a
port
which
is
using
those
PVC
up
and
where
is
it
volume
amount
volume?
Yes,
that's
the
name.
Joseph
test
is
the
name
of
my
PVC
and
as
soon
as
I
start
see,
spot
witch's
mountings
his
assistant
volume,
as
this
PVC,
the
president
William,
was
created
and
also
sees
that
to
change
the
resource
group.
For
that
I
see
that
here
in
the
azure
portal.
B
B
B
Okay,
back
to
the
slide
and
I
stopped
here,
I
think
on
the
slide.
If
something
goes
wrong,
where
I
showed
you
typical
procedures,
you
can
do
if
something
is
goes
wrong
or
is
not
as
expected
on.
The
second
slide,
yeah
I
mentioned
as
a
effect
I
stumbled
over
a
few
times
since
a
few
days
that,
even
if
the
VMS
are
started
and
have
are
configured
by
ignition
and
have
an
internet
connection,
let's
say:
don't
pull
image
from
the
internet.
Maybe
mine,
fresheners
I,
don't
see
anything
in
the
locks.
B
Yeah
I
see
that
some
cubelet
script
is
restarting
all
the
time.
Maybe
something
is
there
is
a
race
condition,
I
don't
know,
but
it
seems
that
it
does
not
run
successfully.
It
doesn't
pull
pots,
and
this
means
it's
yes,
SVM
is
dead,
I,
don't
know
a
procedure
how
to
get
it
back
to
life,
but
I
think
this
will
clarify
in
the
next
and
the
next
time
if
the
Masters
are
affected
by
this
effect,
I
think
you
have
not
much
chance
but
to
recreate
a
cluster
to
delete
it
and
to
create
a
new
one.
B
If
worker
is
affected,
I
got
it
working
one
times.
If
I
deleted
the
worker
in
the
azure
portal,
then,
as
a
machine
API
operator
complains
about
that,
he
doesn't
find
CBM
again
then
I
deleted
a
machine
customer
resource.
That's
a
name
for
four
VMs
in
in
okd
I
deleted
it
and
I
exported
as
a
configuration
from
a
working
machine,
customer
resourceful
working,
VM
and
created
a
new
machine
from
that
and
I
did
this
a
new
worker
VM
was
created
by
by
the
operator
and
everything
work
afterwards
yeah,
it's
it's
some
kind
of
workaround.
B
B
Of
opt
for
currently
is
possible
in
Asia.
Even
it
takes
sometimes
a
few
attempts
yeah,
but
things
get
smoother
and
smoother.
It's
yeah
I,
see
in
the
future
very
optimistic,
yeah,
I
think
I
think
it's
nothing
is
unsolvable.
Unsolvable
unsolvable,
here
story,
my
heck,
I'm
implemented,
is
not
necessary
anymore.
B
If
see
fit
or
cause
image
is
available
on
the
edge
of
marketplace,
but
I
would,
it
would
be
I
would
be
said
if
people
have
to
wait
until
that,
because
I
asked
in
the
community
when
it
will
be
available,
and
the
answer
was
said
as
I
can't
give
ya
a
schedule
about
that.
But
ya
say
surprised
us.
A
lot
in
the
in
the
last
month's
are
said
can
get
very
fast.
B
Sometimes
they
still
are
effects
during
the
installation
that
must
be
observed.
Ok,
D!
If
it
is
installed,
it
seems
to
run
stable
and
yeah
for
sure
it's
a
lot
of
fun.
We
work
with
over
here.
I
love
it
a
lot.
I
can
really
say:
I
met.
My
team
also
always
loves
about
me
that
I'm
so
a
fanboy
about
what
the
software
about
yeah.
It's
so
carefully.
Crafted
piece
of
software
I'll
love
it
and
yeah
hail
to
redhead
and
the
community
for
that.
B
Thanks
to
regime,
Christian,
dusty
and
many
others
for
your
support,
I'm,
always
asking
lots
of
questions
and
slacking
and
my
ticket.
Sometimes
my
tickets
are
not
real
problems
but
based
on
inexperience
from
my
side,
but
yeah
people
always
friendly
and
help.
If
I
can
that's
very
nice,
very
nice
community
allow
it.
B
Point
in
okay,
d3
balls
that,
if
used
image
scanners,
which
are
reporting
vulnerabilities,
also
on
containers
which
are
running
this
software
looks
for
images
where
are
where
they
are
running
containers
and
because
the
images
for
okay
d3
were
rather
old.
Sometimes
we
got
lots
of
problem
reports
and
we
had
to
patch
the
okd
images
on
our
own
because,
yes,
yeah
were
sometimes
no
newer
ones,
and
this
procedure
is
very,
very
bad
for
us,
because
nobody
in
truth
in
this
moment
that
the
images
or
sea
containers
based
on
this
images
run
properly
I.
B
B
There
would
be
no
new
image
for
that.
Where
you
can't
do
could
do
a
yum
update,
yeah
and
which
and
rural
said
you
get
hot
fixes
at
least
yeah,
which
lands
into
the
image.
It
would
be
absolutely
if
there
would
be
updated
images.
This
would
have
big
advantage
over
okay
d3,
where
we
suffered
a
lot
in
the
last
months
on
yeah.
This
customly
patched
images
yeah
Gloucester.
B
If
s
was
the
next
next
point,
it
was
very
unstable
in
in
okay
d3
we
removed
it
from
the
cluster,
but
as
far
as
I
understand,
ok
d4
is
a
completely
new
storage
solution
which
was
proposed.
Okay,
I
think
it's
based
on
seven
rook,
it's
very
nice
yeah
then
we
have
one
l-dub
swing,
creates
a
lot
of
traffic
on
our
Active
Directory
servers
in
okay
d3,
because
we
have
nested
group
queries
and
the
colleagues
asked
me
to
tell
ya
to
tell
that.
B
Maybe
maybe
there's
something
we
don't
know
about.
Ok,
d4
already
available,
I'm
said,
logging
stack
and
okay
III
was
was
rather
horrid.
Elasticsearch
was
I,
think
it's
version
5.
Apparently
we
have
I,
think
elastic
7
right,
don't
understand.
Why
is
there
is
no
newer
version,
maybe
because
it's
decoupled
from
ok
G?
B
That
would
be
ok
now,
currently,
it's
not
possible
by
the
way
to
install
logging
stack
because
in
the
documentation
there
is
a
section
about
how
you
can
install
it
and
as
far
as
remember
it
should
be
taken
out
of
C
operator
app,
but
there
is
no
logging
solution
in
currently
in
the
operator
hub.
It
seems
as
if
it
as
was
filtered
out
during
a
c
c
procedure,
as
the
only
C
community
operators
got
ya
filtered
out
for
okt.
Formerly
there
were
also
a
redhead
project
present
and
now.
A
B
Are
not
anymore
present
next
point
documentation
about
over
shifts
for
okay,
DS
internal
attack.
Sure
would
be
nice
as
I
understand
that
it's
planned
after
okay
D
is
GA
that
the
community
is
asked
to
work
gently
on
okay,
D,
that's
a
point
where
it's
absolutely
necessary,
in
my
opinion,
to
have
more
high
tech,
chure
docks
about,
let's
say
purpose
of
operators
diagrams
how
things
play
together,
but
I
think
it's
it's
nothing.
You
yeah!
It
will
come
I'm,
absolutely
sure,
but
it's
also
necessary.
B
If
you
want
to
have
some
unity
with
you
insert
a
little
bunch
more
a
debugging
features
would
be
nice.
During
my
presentation.
I
mentioned
a
few
one.
Maybe
it's
possible
to
have
private
keys
for
masters
and
workers
in
the
bootstrap
p.m.
so.
If
things
go
wrong,
that
I
don't
have
to
get
them
public
IP
first
manually,
because
it's
yes,
it's
if
you
do
that
over
and
over
it's
the
can
waste
your
time
with
that
and
yeah
I
think
that
was
a
moment
what
we
would
like
to
be
improved
in
okt.
B
Here
is
my
last
slide:
some
links
about
valuable
information
sources.
Most
important
informations
was,
in
my
opinion,
where
I
got
the
most
feedback
for
development.
Is
the
slack
channel
overshift
deaf
in
the
kubernetes
workspace,
as
there
is
a
situation
that
yeah
I
ask
something
and
normally
takes
a
few
minutes
and
someone
answers
even
my
colleagues,
which
they
always
open
the
github
issues
and
wondering
sometimes
was
there
is
no
response,
as
his
colleagues
were
absolutely
amazed
about
the
response
times
on
slack
yeah.
B
It's
is
this
where
the
great
community
shows
its
power,
because
you
always
get
valuable
information.
Parent,
there's,
always
a
nice
atmosphere.
This
is
Astrid
yeah,
send
we
have
do
Kitty
up
shift
or
Katie
link
here.
You
should
put
your
issues
inside
here.
Not
in
overshift.
Installer
was
the
other
repositories
because
yeah,
if
things
maybe
come,
the
problems
come
from
misunderstanding:
okay,
D
it
can
be
solved
here
and
we
don't
bother
so
developers
about
that.
B
Please
new
issues
here,
then
we
have
C
original
release,
a
page
where
all
and
new
releases
are
announced-
and
you
see
a
very
cool
change,
lock
its
allows
the
presentation
on
this
page.
It's
so
completely
different
to
what
we
are
used
to
the
former
versions.
It's
absolutely
great
I
can
absolutely
encourage
you
to
visit
this
page.
I
love
it
yeah
I
could
I
could
watch
it
all
the
night
yeah
because
says
yeah,
it's
a
it's
queer,
so
we
have
next.
B
We
have
the
okie
working
group
page
and
there
you
have
information
about
COPD
working
groups,
the
attempt
it
occurs,
be
weekly
and
normally
on
I
think
Tuesday
evening
in
German
time
series
also
a
calendar
for
that.
You
can
follow
this
link
and
you
will
get
the
entry
in
your
your
calendar
automatically.
We
have
Co
per
shift
documentation
few
days
ago
also
see
Okeke
pre
do
documentation
got
released,
so
you
also
should
visit
it
and
if
you
have
Corrections,
if
you
find
some
errors,
you
can
I
missed.
Authority
I
forgot
the
link.
B
There
is
also
repository
under
open
shift,
open
shove,
Docs
I
think
you
can
open
the
pull
request
board
for
your
documentation
box.
There
don't
forget
to
put
the
to
put
the
ok
D
in
Z
between
brackets
in
the
title,
so
the
documentation
people
can
filter
out.
What's
special
for
what
has
to
be
separated
from
open
shift,
okay,
D
and
the
last
one
say
there
are
two
ripples
they
are
currently
in
this
phase,
as
in
most
differences
between
overshift
and
okay,
D
are,
as
far
as
understood.
This
is
help
shift,
install
an
operative
machine.