►
From YouTube: Kubernetes UG VMware 20200903
Description
September 3 2020 meeting of the Kubernetes VMware User group - including coverage of the Helm chart for deploying the vSphere CSI storage plugin and Troubleshooting a deployment of Kubernetes on vSphere
A
So
I
posted
links
to
the
agenda
doc.
I
was
going
to
do
a
a
little
bit
of
material
on
troubleshooting
running
kubernetes
on
top
of
the
vsphere
hypervisor,
I've
got
a
deck
ready.
I
apologize.
A
We
I
we
had
a
speaker
cancellation
just
a
few
days
ago
and
I
was
on
pto,
so
I
kind
of
rushed
this
together,
but
I've
got
a
little
something
and
then
miles
was
going
to
talk
about
some
recent
work
on
a
health
chart
for
installing
the
csi
storage
plug-in
that,
in
a
way
that
interoperates
with
the
cloud
provider
plug-in
that
david
von
fenin,
implemented
and
david
is
on
the
air
too,
and
feel
free
david
to
chime
in
on
any
comments
or
remarks
you
want
to
make.
B
Yeah,
no
I'll
I'll
go
first
because
mine's
fairly
concise
anyway.
So
let
me
just
share
my
browser.
Oh
host,
disabled.
C
B
Awesome:
let's
try
that
again,
cool
safari.
Can
you
see
that
yes,
okay,
so
cormac
and
myself
have
received
a
lot
of
feedback
from
fields?
Saying
csi
is
too
hard
to
install
or
not
really
too
hard,
but
too
error
prone
very,
very
easy
to
mess
it
up.
So
we
asked
engineering,
you
know:
is
there
a
helm
chart
on
the
to
sort
of
go?
Is
it
is
it
a
priority?
B
Whatever
they're
working
on
an
operator
and
so
helm
chart
comes
secondary
to
the
operator,
so
they're
they're
focused
on
that
and
we
thought
well
how
hard
can
it
be?
So
we
implemented
a
helm
chart
for
csi
ourselves.
It
also
installs
the
cpi,
as
well
so
david
wrote
a
helm
chart
for
the
cpi
a
while
back.
B
This
is
the
csi
helm
chart
whenever
you
install
it
lists
that
as
a
prerequisite
or
as
a
dependency,
so
it'll
automatically
install
the
cpi
for
you
as
well.
So
if
you
vanilla,
kubernetes
cluster,
this
makes
it
way
easier
to
get
up
and
running.
Actually
so
at
the
minute
it
only
deploys
csi
version
2.0,
which
is
backwards
backwards,
compatible
with
67u3
as
well,
and
it
has
all
of
the
new
support
around
read,
write
many
volumes
and
all
that
good
stuff.
So
basically,
we've
set
up
something.
B
We
expect
this
to
be
moved
away
from
cormic's
github,
maybe
in
the
next
two
weeks
or
so
once
we
get
the
pull
request
opened
into
the
csi
driver
repository
so
that
it
can
be
hosted
somewhere,
that's
sort
of
vmware
official
and
not
just
you,
know,
a
personal
github.
So
at
the
minute
you
add
cormec's
chart
repository
you
do
a
repo
update,
then
there's
just
some
stuff
around
testing
it.
B
If
you
want
to
test
it,
how
you
do
the
template
and
see
what
it's
going
to
install,
but
the
actual
sort
of
bits
and
pieces
that
make
it
work,
here's
the
sort
of
required
config.
So
you
can
see.
Config.Enabled
is
true.
The
vcenter
username
password
for
the
vcenter,
the
data
center
in
the
vcenter,
and
then
the
cluster
id
says
change
me.
B
So
just
you
fill
that
in
with
whatever
string
you
want,
and
currently
we
also
have
to
duplicate
the
same
information
as
we
have
here
for
the
cpi,
because
it
requires
the
same
information
but
the
way
the
helm
charts
are
written.
B
They
we
can't
sort
of
use
the
same
parameter
for
both.
So
I
did
some
changes
and
opened
the
pr
with
david's
helm,
chart
that
expose
global
variables,
and
the
idea
is
that,
instead
of
having
config.vcenter
and
vsphere
cpi.config.vcenter,
we
just
have
a
global.config.vcenter
and
then
that'll
apply
to
both
things,
because,
generally,
what
we've
seen
people
do?
Is
they
they
just
install
a
cpi
and
csi
into
the
exact
same
environment
with
the
same
config?
B
Very,
very
rarely
do
they
change
so
we're
going
to
leave
these
in
here
as
overrides,
and
the
idea
is
that
we'll
set
a
global
so
that
you
just
can't
mess
the
config
up
from
one
to
the
other.
So
if
you
run
this
it'll
go
ahead,
install
the
cpi
install
the
csi
and
it'll
come
up.
It's
really
that
simple.
A
I
hope
you
don't
mind
if
I
interrupt
you,
but-
and
maybe
I'm
gonna
ask
put
david
on
the
spot
here
too,
for
this
duplication,
I'm
a
little
rusty
in
hell
charts,
but
I
think
that
you
should
be
able
to
set
these
using
environment
variables
in
a
way
that
you
could
write
a
bash
script
or
something
reuse
it
in
kind
of
one
pass
right,
because
I
mean
these
set
commands
could
just
pull
something
out
of
an
environment
variable
and
if
I'm
not
mistaken,
there's
a
way
to
feed
these
in
through
yaml.
D
B
A
So
I
don't
know
it's,
maybe
beyond
the
scope,
we're
just
pointing
out
here
in
this
meeting
that
the
facilities
available,
but
I
think,
there's
opportunity
to
embellish
this
later
and
maybe
we'll
get
some
blog
posts
out
or
I
think
miles,
and
I
are
planning
a
talk
at
kubecon
north
america
later
this
year.
So
maybe
we
can
have
a
more
polished
version
of
this
prepared
for
that.
B
B
It's
not
it's,
probably
not
as
robust
as
it
should
be,
but
it
does
the
job
because,
as
far
as
csi
is
concerned,
anyway,
the
configuration
is
really
very
simple:
it's
just
the
the
secret
that
needs
updated,
so
it
was
pretty
low
hanging
fruit.
All
the
other
niceties
around
this.
You
know
that
that
is
stuff
that
you
know
where
we
would
welcome
any
feedback
that
you
guys
have
on
this
yeah.
I
guess
the
good
thing.
A
About
this
being
early
stages-
and
you
know
still
on
informative
stages-
is
that
if
any
users
are
out
here
either
live
in
the
meeting
or
watch
the
youtube
video
later.
Please
let
us
know
now,
because
there
it's
often
much
easier
to
do
these
before
you
get
the
official
version
one
out
there
and
have
the
documentation
locked
down,
because
changes
then
require
a
bunch
of
edits
and
pr's
and
all
that-
and
maybe
we
can
get
it
right
on
the
first
pass
with
the
right
amount
of
feedback.
That'd
be
nice.
Wouldn't
it.
B
So
yeah
we've
just
got
an
example
command.
You
know
if
this
wasn't
clear
enough,
you
know
here's
the
actual
values
that
you'd
want
to
fill
in
and
then
we
have
another
one.
If
you're
using
vsan-
and
you
want
vsan
file
services
support,
then
we
fill
that
out
with
how
you
set
up
the
extra
config
for
vsat
file
services
as
well
so
and
then
the
uninstall
so
really
pretty
simple.
It's
out
there.
A
And
just
in
case
for
you
newcomers,
if
you
weren't
aware
of
the
functioning
of
this
group,
we're
prepared
to
support
running
kubernetes
on
all
we're
prepared
to
support
running
all
forms
of
kubernetes
on
vsphere,
so
it
doesn't
have
to
be
a
vmware
distribution.
It
could
be
open
shift,
it
could
be
rancher.
It
could
be
that
you
roll
your
own
right
off
the
github
repository,
but
this
group
operates
under
the
auspices
that
we
support
all
forms
of
kubernetes
on
the
vsphere
platform.
B
Was
just
one
thing
I
wanted
to
add
after
you
said
that
this
the
the
helm
chart
and
its
current
iteration?
I
know
it
doesn't
work
on
red
hat
openshift,
because
I've
tried
it
because
they
require
a
specific.
I
think
they
call
it
security
context,
config
scc,
and
we
don't.
We
just
haven't
built
out
the
ammo
for
that.
Yet
so
I
I
know
it
doesn't
work
on
red
hat
openshift,
but
it
does
work
on
stuff
that
doesn't
have
like
a
any
other
additional
requirements
around
security,
or
anything
like
that.
A
Lot
of
these
distributions
already
have
their
own
polished,
installer
that
take
these
kind
of
inputs,
and
if
I
were
a
user
of
those
or
vmware's
some
of
vmware's
kubernetes
distributions.
Do
this
too.
I'd
go
with
the
documented
path,
because
if
you
end
up
going
into
a
support
engagement-
and
you
tell
them-
you
did
something
different
from
the
one-
that's
in
their
standard
workflow
in
the
documentation-
it's
probably
just
going
to
slow
things
down,
they
might
ask
you
to
do
it
over
again
anyway.
So
maybe
this
kind
of
thing
is
more
useful.
A
I
know
I
play
around
with
these
using
this
in
home,
lab
scenarios
and
things
like
that
and
a
lot
of
people
just
getting
started
aren't
prepared
yet
to
even
make
a
selection
of
a
commercial
distribution.
So
I
think
that's
the
prime
audience
for
something
like
these
tools
and,
like
I
say
many
of
those
distributions
are
already
going
to
have
the
equivalent
of
this
or
maybe
even
better,
in
terms
of
validating
parameters,
as
you
put
them
into
whatever
ui
or
other
mechanism.
They
have.
F
I
saw
it
tweeted
out
and
then
tested
it
out
this
morning.
Okay,
cool.
A
Okay,
boy:
I'm
really
glad
you
joined
us
today
and
gave
us
that
feedback
that
that
helps
a
lot.
D
Miles
up
for
the
the
pod:
are
you
talking
about
the
pod
security
context?
Yes,
I
am
yeah
yeah.
So
actually,
if
you
want
an
example
of
that,
that's
actually
implemented
in
the
cpi
chart,
so
you
might
be
able
to
just
like
oh
perfectly,
lift
and
shift
it
over.
So
that
would
be.
B
D
It
definitely
has
the
you
know
like
if
you,
if
you
want
to
enable
it
or
not
it,
has
that
flag,
and
then
it
you
know,
has
a
lot
of
the
stuff.
That's
like
templated
out
where
you
can
also
override
usual,
like
values
that
most
people
kind
of
want
to
override,
so
it,
but
it's
there
for,
for
you
to
kind
of
take
a
look
at.
If
you
want.
B
A
A
A
So
forgive
me
if
there's
any
egregious
typos
in
here,
but
let
me
give
this
a
try
and
I
had
aspired
to
doing
a
live
demo
of
some
of
these
things
that
I
put
in
this
powerpoint,
but
I'm
traveling
at
the
moment
family
related
matter-
and
I
found
that
I'm
having
to
connect
through
two
layers
of
vpn
and
I
can't
actually
get
command
line
on
some
of
the
things
I'd
like
to
right
now.
So
no
live
demo,
but
I
tried
some
of
what's
in
these
slides
before
I
left.
A
A
And
let
me
throw
this
into
presenter
mode
and.
A
Okay,
so
this
is
an
intro
to
troubleshooting
on
vsphere.
What
do
you
do
when
things
go
wrong?
Just
kind
of
where
do
you
go
to
find
your
log
files
is
what
this
amounts
to
so
I'll
start
with
a
brief
overview
of,
what's
involved,
with
running
kubernetes
on
vsphere.
A
Maybe
this
is
too
elementary
for
our
audience,
but
we
do
get
some
people
who
watch
later
on
youtube.
So
forgive
me
if
this
is
kind
of
2-101
for
any
of
you,
but
I
think
it
might
help
some
people,
then
I'm
going
to
cover
where
this
applies
and
where
it
doesn't,
because
I'm
skipping
some
kind
of
older
platforms
and
releases
and
then
finally
leave
with
resources.
A
So
there's
two
parts
here:
the
cloud
provider
being
the
first
one,
and
this
is
an
interface
that
allows
kubernetes
to
abstract
out
the
complexity
and
variances
of
different
hosting
platforms.
You
know
vsphere
is
one
when
you're
running
on-prem,
but
public
clouds
would
be
the
other
major
hosting
platform,
anything
from
amazon
aws
to
google
cloud
or
azure
cloud.
The
cloud
provider's
goal
is
to
make
those
look
close,
similar
enough
that
you
can
design
applications
that
are
portable
and
can't
tell
and
don't
care
where
they're
running.
A
The
key
element
of
this
cloud
provider
is
the
node
abstraction
and
that's
what
provisions
and
integrates
in
the
kubernetes
nodes
on
vsphere.
That
would
be
the
vms
that
act
as
kubernetes
cluster
nodes.
A
value.
Add
that's
unique
to
the
vsphere
cloud
provider
is
that
it
does
support
respecting
availability
zones
that
are
implemented
in
the
underlying
infrastructure
and,
at
the
same
time,
tying
that
in
with
storage,
there
are
some
optional
features
of
cloud
providers,
handling,
routing
and
load.
A
A
A
If
you
went
back
to
2016
or
2017
2018-ish,
there
were
entry
plug-ins.
That
is
something
that
is
out
there
still
available,
but
deprecation
warnings
have
already
been
issued.
There's
both
entry
storage
plugins
as
well
as
an
entry
cloud
provider.
They
still
are
supported,
however,
they've
already
gone
a
little
over
a
year
with
no
feature
enhancements.
A
So
this
what
they're
in
a
phase
where
they're
getting
security
patched,
but
all
new
features
are
going
into
the
auditory
variants
of
this,
and
today,
I'm
only
going
to
cover
troubleshooting
using
the
out
of
tree
tools,
there's
in
a
new
deployment,
there's
pretty
much,
no
reason
to
use
those
entry
variants,
the
only
one.
A
C
B
Well,
the
entry
one
is
actually
both
in
the
one
thing,
so
the
vcp
covers
storage
and
and
the
cloud
provider
current
planner
record
looks
like
1.21,
which
I
think
is
the
beginning
of
next
week.
Next
year
it
moves
around
a
little
bit.
There's
some
talk
of
1.22,
but
it
basically
the
hard
requirement
for
it
to
be
removed.
Is
that
vcp
to
csi
migration?
Is
there
and
stable?
So
when
that
happens,
then
it'll
be
removed.
A
Yeah
it.
This
has
been
something
where
I'm
a
little
skeptical,
because
they've
thrown
out
times
that
have
always
been
essentially
any
time
in
my
memory.
It's
been
we'll
do
it
next
year
and
then
the
new
year
comes
around
and
then
it's
still
the
next
year
after
that,
so
I
think
you're
definitely
safe
through
the
end
of
this
year,
and
a
lot
of
it
depends
on
how
much
you
believe
these.
A
Whether
these
dates
are
hard
numbers
or
wishful
thinking,
and
I
guess
I
don't
want
to
go
on
the
record
of
my
personal
opinion,
but
I
think
you're
definitely
safe
through
the
end
of
this
year
and
when
it
comes
about
once.
One
thing
to
keep
in
mind
for
vsphere
is
that
the
two
are
tied
together,
because
we,
the
out
of
tree
cloud
provider,
isn't
designed
to
work
within
tree
storage.
It
kind
of
might,
but
I
don't
think
it's
tested
and
I
wouldn't
recommend
it
so
that
you're
going
to
likely
flip
the
two
of
them.
A
At
the
same
time,
in
the
same
act,
there
is
a
migration
plan
for
the
storage
that
I
believe
is
destined
to.
Allow
you
they,
they
will
replace
the
current
entry
storage
driver
with
something
that
passes
through
calls
to
the
out
of
tree
one,
and
that
solution
should
handle
the
storage
angle.
But
I'm
not
so
sure
about
the
cloud
provider
angle.
B
Yeah,
I
can
give
you
a
little
bit
more
detail
on
the
storage
thing,
so
that's
the
vcp
to
csi
migration
that
I
was
referring
to.
We
essentially
take
the
vcp
calls
and
it
becomes
just
like
a
shim
layer
and
all
those
calls
get
passed
on
to
the
csi.
There's
some
internal
discussion
at
the
minute.
Obviously-
and
we
translate
stuff-
don't
support
it
in
vcp
that
isn't
supported
in
csi,
for
example
like
raw
spvm
parameters
that
were
in
pcp,
not
in
csi.
B
So
how
do
we
support
and
translate
those
into
the
the
new
way
of
doing
things?
So
there
is
some
complexity
around
it,
but
the
idea
is
stable
is
that
it
will
be
a
completely
transparent
operation
that
will
change
the
backend
disk
types
from
vmdks
to
fcds,
and
you
know
does
all
of
that
stuff
for
you.
So
it
should
be
something
that
you,
basically
just
don't
notice.
B
C
Yeah
we
were
just
talking
about
this
this
morning.
We've
just
had
I
mean
covets
thrown
off
our
upgrade
plans
for
the
whole
year
of
things,
and
so
I
mean
we're
on
a
lot
of
places
where
six
seven
u2
but
we're
trying
to
get
to
u3
and
now
it's
like
well,
maybe
we'll
just
go
to
7-0,
but
then
you've
got
to
do
testing
for
that,
and
then
you
still
have
to
a
lot
of
places
to
roll
out
too.
So
we're
just
trying
to
see
what
we're
we're
continually
upgrading
to
the
newer
kubernetes
versions.
C
A
So
if
you've
got
any
opinions
on
the
matter,
we're
prepared
to
take
those
under
advisement
too.
You
know
we're
we're
always
looking
for
feedback
from
users,
so
feel
free
if
whatever
you've
heard
or
if
you've
got
hard
limits
or
difficulties
that
would
be
caused
by
some
date.
Let
us
know
and
we'll
see
what
we
can
do
about
it,
the
other
thing
to
throw
out
there
with
regard
to
any
systems.
You
do
have
to
upgrade
that
migration
plan.
A
B
Right
so
currently,
the
plan
for
the
migration
stuff
is
the
kate's
bit
is
going
to
be
compatible
with
119
and
above
so
that
will
that
is
beta
or
will
be
beta
in
1.19,
so
that
that
part
will
be
there.
It'll
require
the
next
version
of
vsphere,
whatever
number
we
choose
for
that,
but
we
are
in
discussions
as
to
earlier
versions.
You
know
how
do
we
support
earlier
versions
of
the
platform,
whether
in
a
patch
release,
or
otherwise
we
haven't
made
any
firm
decisions
on
that
current
plan
of
record?
B
Is
it's
going
to
require
a
new
version
of
vsphere,
but
we'll
see
if
that
changes
based
on
feedback
and
we've
received
a
lot
of
feedback
to
say
it
should
change.
So
if
you
have
more
feedback,
let
us
know.
A
Okay,
unless
you've
got
anything
else
on
that
subject,
I'll
advance
my
slide
deck.
So
with
regard
to
the
cloud
control
manager,
this
you
know
it's
the
interface
to
the
the
vsphere
hypervisor,
that's
hosting
the
vm's.
Acting
is
kubernetes
nodes.
A
The
there
is
going
to
be
a
cloud
controller
manager,
pod.
Typically,
there's
going
to
be
one.
There
may
be
mechanisms
to
deploy
more
than
one,
but
one
is
going
to
be
it's
going
to
use
leader
election,
I
believe,
is
the
way
it
works.
So
you
can
cube
cuddle
in
with
admit
credentials,
get
the
pods
in
the
system
name,
space,
some
people's
distributions
or,
if
you're
rolling
your
own
from
open
source.
A
Maybe
you
could
have
put
this
in
a
name
space
other
than
cube
system,
but
I
think
it's
typically
in
cube
system
get
the
list
of
of
pods
just
grep,
for
the
word
cloud
usually
is
going
to
work.
Maybe
you'll
get
a
false
positive,
but
for
me
and
my
system,
this
work
gets
you
a
pod
name.
A
Then
you
can
do
cube
cuddle
logs
with
the
name
of
that
pod
just
say
all
containers
and
that
will
dump
your
logs.
What
you'd
expect
to
show
up
there?
You
know
generally
the
reason
most
people
would
even
be
concerned
with
getting
logs.
Is
that
something's
not
working
right,
but
you
can
see
the
parsing
of
the
configuration
file
at
the
very
beginning
of
those
logs
and
if
you
had
any
issues
related
to
an
element
of
that
configuration
being
bad
or
an
authentication
problem
on
whatever
account
credentials,
you
use
being
bad.
A
You
could
expect
it
to
show
up
there
with
a
little
bit
of
detail.
That
should
be
enough
to
get
you
started
as
to
maybe
what
line
items
are
at
fault
in
your
configuration
or
adding
some
additional
details
when
you
go
seek
help
in
slack
through
your
vendor,
whatever
you,
whatever
path
you
choose
to
go,
does
anybody
else
have
anything
more
to
add
on
the
on
troubleshooting
the
vsphere
cloud
provider
and
things
that
they
found
through
the
logs?
That
might
be
helpful
to
others.
A
So
I
was
like
no,
but
anyway,
this
is
a.
This
is
a
decent
starting
point.
The
the
docs
do
have
some
material
on
this,
but
you
know
they
maybe
have
advisories
like
go
get
the
logs
and
it
sometimes
beginners.
Could
you
could
even
use
a
little
guidance
on
the
cube
cuddle
commands
for
even
getting
logs?
You
know
we've.
I
know
we
have
people
out
there
who
are
absolutely
rookies,
trying
to
learn
their
way
up
kubernetes.
A
So
forgive
me
if
this
is
a
little
too
elementary,
but
I
think
it
it
should
be
helpful
to
some
people.
A
So
you
can
see
the
volumes
getting
mounted
into
the
vms
in
the
recent
tasks
window
of
the
ui
and
then
with
vsphere
itself.
I
think
it's
fair
to
say
that
most
people
use
a
log
gathering
solution,
whether
it
be
log
insight,
wavefront
or
taking
it
to
splunk
or
whatever
pretty
much
vsphere
will
allow
consolidation
of
logs
to
whatever
you
choose,
and
sometimes
it's
helpful
to
look
at
things
from
the
vsphere
perspective.
A
If
it's
just
getting
your
kubernetes
cluster
off
the
ground,
I
think
the
kubernetes
logs
are
more
likely
to
be
your
first
step,
but
if
it's
something
like
intermittent
connections
or
logs
or
volume
outs
that
won't
go
away,
what
you
would
find
in
the
vsphere
logs
may
be
a
useful
adjunct
to
this
anything.
You
want
to
add
on
that
with
regard
to
that
miles,
or
does
this
look
good.
B
No,
I
think,
that's
pretty
good.
The
other
useful
part
of
the
ui
would
be
if
you're
using
csi
would
be
go
into
the
cluster
level
and
then
monitor
and
then
there's
the
container
cloud
native
storage
ui
that
we've
got
there.
That
shows
you,
the
pvs.
What
they're
mounted
into
you
know
what
the
storage
class
is
the
compliant
all
that
good
stuff.
So
you
can
see
all
of
the
infrastructure
and
the
kubernetes
layer
stuff
in
there.
A
Yeah
and
I
apologize
for
doing
kind
of
this
lousy
screenshot,
but
I'm
stuck
in
a
hotel
room
through
double
vpns,
and
I
couldn't
get
back
to
my
vsphere
to
do
a
live
demo
of
this,
like
I
had
hoped
so,
the
moving
on
to
the
csi
storage
plug-in
and
how
you
would
troubleshoot
by
getting
logs
there
first.
The
first
step
is
to
actually
find
the
pods
that
are
related
to
csi
and
that
line
in
yellow
cube
cut
will
get
pods
all
name.
Spaces
grabbing
for
csi
is
generally
going
to
work.
A
It's
kind
of,
unlike
it's
a
pretty
broad
pattern
net,
but
I
don't
think
it's
likely
that
you'd
have
any
other
pods
running
with
csi
in
the
name.
So
this
this
would
generally
work
for
me
and
you're
going
to
find
that
the
pods
are
going
to
have
multiple
containers.
A
Depending
on
what
issue
you
have,
you
could
dump
all
the
logs
from
all
the
containers
at
once,
which
is
that
cube
cuddle
logs
you've
got
the
name
there.
So,
on
my
system,
my
name
was
vsphere
csi,
controller,
hyphen,
zero
and
then
saying
all
containers
you
might
have
as
many
as
five
containers
in
there
that
have
different
functions,
so
you
can
isolate
them
by
one.
If
you
know
your
problem
is
scoped
down
to
one
particular
area.
A
There's
the
vsphere
csi
controller,
the
attacher,
the
sinker,
the
provisioner,
and
they
all
have
different
roles
in
the
phase
of
how
this
thing
goes
down
through
the
phases
of
allocating
a
persistent
volume
claim,
creating
a
persistent
volume
syncing
between
the
the
kubernetes
world
and
the
vsphere
world.
A
B
Just
just
a
little
bit
yeah
in
my
experience,
the
only
one
that
you'll
get
useful
logs
out
of
as
in
if
you're,
trying
to
troubleshoot
volume,
provisioning
or
something
like
that
is
the
vsphere
csi
controller
container,
attacher
provisioner,
all
that
stuff
is
usually
100
and
the
vsphere
syncer
is
basically
just
a
metadata
syncer.
All
it
does
is
pulls
labels
and
tags
and
stuff
like
that
from
kubernetes
and
syncs
them
into
the
vsphere
ui.
So
it's
not
in
the
control
path
of
actually
provisioning
volumes
and
stuff.
A
Okay
good
to
know-
and
I
think
there's
one
other
one-
that's
pretty
much
always
going
to
be
some
of
these-
don't
even
they're
not
even
unique
to
vsphere,
because
the
csi
project,
I
think,
provides
some
of
these
side
cars
just
as
a
generic
implementation
across
all
storage,
all
flavors
of
storage
and,
I
believe,
there's
some
sort
of
a
keep
alive
one.
There
too.
A
That
is
just
the
health
check
that
this
this
pod
is
actually
running,
but
unless
things
are
totally
hosed,
I
wouldn't
expect
that
you'd
find
any
useful
troubleshooting
information
by
looking
at
the
logs
of
that
container.
But.
B
Yeah,
the
only
two
that
we
actually
provide
are
the
ones
prepended
with
vsphere
dash,
so
there
are
only
two
containers
that
we
actually
ship,
vsphere,
csi-controller
and
vsphere
sinker.
All
the
other
containers
that
run
as
part
of
that
pod
are
all
provided
by
upstream.
A
A
So
I
don't
know
what
happened
here.
I
had
some
more
slides.
Oh
here
we
go.
I
guess
my
laptop
is
just
slow
here.
A
Okay,
okay,
so
here's
some
additional
resources.
This
is
just
you
know.
In
addition,
obviously
you
should
read
the
docs
and
the
docs
are
the
bottom
of
this
list,
but
I
did
find
a
for
people
moving
from
entry
to
out
of
tree.
There
was
a
good
blog
post
by
eric
shanks
and
it
actually
had
cut
and
paste
examples
of
a
configuration
entry
and
comparing
it
to
one
out
of
three
miles
and
cormack
hogan
who's,
not
on
the
call
are.
A
I
just
recommend
their
blogs
for
just
about
everything
and
I
didn't
put
them
here,
but
those
are
great
resources
just
over
all
time.
There
is
a
deep
dive
here
on.
This
is,
from
the
kubernetes
perspective,
on
the
principles
that
went
into
moving
the
cloud
providers
from
entries
out
of
tree,
and
this
really
is
a
deep
dive
like
I
don't
think
the
average
user
has
to
go
there,
but
if
you
really
want
an
in-depth
that
would
allow
you
to
troubleshoot,
maybe
the
most
difficult
problems
and
explain
what
some
of
the
terminology
is.
A
A
The
one
there
that
I
remember
was
that
there's
potentially
a
race
condition
on
involved
with
deleting
a
persistent
volume
compared
to
deleting
of
a
vm
that
it
might
be
attached
to
that.
Maybe
is
something
to
watch
out
for
triggered
by
deleting
a
whole
name
space
at
a
time
there
were
a
few
other
issues
on
that
list
and
that
list
probably
is
gonna.
You
know
it's
it's
maintained,
so
it
might
change
over
time,
but
that's
something
to
be
aware
of.
A
B
No
that
that
about
covers
it,
really
I
I
would
just
say
if
you
do
have
storage
problems.
It's
usually
you
haven't,
enabled
node
underscore
uuid,
which
is
a
prerequisite
it's
it's
listed
in
the
docs,
but
usually
that's
the
problem,
and
if
it's
not
that
username
and
password,
it's
probably
wrong
for
virtual
center.
So
it's
usually
one
of
those
two
things
or
you're,
not
running
it.
On
6.7,
u3
or
above
as
we've
had
recently
a
couple
of
people
saying
hey.
A
Okay,
good
advice:
I'm
going
to
stop
the
share,
yet
I
see
bryson
wanting
a
link
to
the
agenda,
so
let
me
throw
it
in
there.
Just
gonna,
appraise
it.
A
The
odd
thing
about
slack
or
about
zoom
is
that
if
you
join
late
and
somebody
already
put
something
in
chat,
I
don't
think
it
your
own
window
doesn't
show
it.
B
I
think
he's
asking
for
the
link
to
the
slides
to
be
added
to
the
agenda,
so
they
can
look
at
the
slides
afterwards.
A
Yeah,
okay,
I
will
do
that.
I,
I
was
just
editing
those
slides
up
to
five
minutes
before
the
meeting,
so
I
didn't
quite
get
it
in
there,
but
I'll
I'll
get
it
in
by
the
end
of
the
day,
and
maybe
I
I
noticed
a
couple
typos
while
I
was
presenting,
so
I
think
I'll
clean
those
up
or
record
the
link
for
all
prosperity
right
now
the
link
is
just
sitting
on
the
hard
disk
on
my
laptop,
so
I
have
to
upload
it
to
a
a
google
drive
location
in
order
to
share
it.
A
Okay,
I
think
that
was
it
for
today's
agenda,
but
we're
open
to
any
other
questions
or
freeform
discussion.
So
one
last
thought
that
I
should
have
put
in
the
troubleshooting
slide,
but
if
going
to
those
logs
doesn't
help
you
they're
we're
prepared
to
accept
questions
and
support
requests
in
these
slack
channels,
there
is
a
slack
channel
unique
to
the
cloud
provider.
I
don't
think
there
is
one
unique
to
the
storage
provider
for
vsphere,
but
there
is
also
a
link,
a
vmware
user
group.
A
Slack
channel
go,
I
guess
I'll.
I
don't
know
if
this
is
official
or
whether
the
people
who'd
respond
to
storage
help
requests
have
a
preference
for
where
it
goes,
but
I
think
that
it
will
get
noticed
if
you
put
it
in
the
user
group
channel
or
the
cloud
provider
channel
do
either
of
you
guys
doing
this
david
or
miles
have
a
preference
for
where
users
should
put
these.
B
A
Okay,
does
anybody
else
have
any
questions
comments?
Remarks,
even
suggestions
for
what
we
address
next
month
are
open.
A
Other
things
that
miles-
and
I
are
open
to
suggestions
for
is,
I
think
we
have
a
deadline
for
getting
a
an
agenda
done
for
the
maintainer
track
session
at
kubecon
north
america,
which
goes
on
the
air
sometime
in
november.
But
if
any
of
you
users
are
out
there
and
have
topics
you'd
like
to
see
discussed
in
detail,
throw
it
out
there
if
you
can't
think
of
it,
now,
just
throw
it
out
there
on
the
slack
channel
and
let
us
know
in
the
next
few
days.
B
E
Okay,
I
have
one
question
actually
or
a
subject
that
I'm
not
sure
if
it
has
been
talked
about
in
a
previous
one,
but
is
the
whole
subject
of
backup
in
kubernetes
by
vmware,
whether
it
has
to
do
with
things
like
a
valero
tool
or
datrium
or
any
of
the
castings
out
there.
You
know
a
little
bit
of
maybe.
A
A
There
there's
two
phases
to
this:
where
we've
got
some
engineers
who
are
trying
to
make
this
open-ended
and
develop
an
infrastructure
to
support
things
like
snapshots
on
volumes
that
enable
kind
of
there's
a
history
of
lots,
of
backup
vendors
that
have
been
out
there
for
decades
now,
and
we
really
have
a
goal
to
enable
all
of
them.
A
You
know
there
are
some
backup
solutions
that
are
kubernetes
unique,
but
they
also
can
benefit
from
an
underlying
infrastructure
that
would
do
things
like
snapshots
and
kiessing
of
applications
running
inside
containers,
which
in
turn
are
inside
vms,
if
you're
on
vsphere
and
we've
got
a
long
built-up
infrastructure
of
that,
I
found
that
most
users
who
went
down
a
path
of
using
a
particular
vendor
for
backups
are
kind
of
loath
to
switch,
because
maybe
if
a
new
vendor
comes
around
long
and
has
some
really
cool,
kubernetes
unique
features.
A
A
So
if
you
were
to
gather
up
a
new
vendor
to
handle
only
your
kubernetes
you're
still
going
to
have
to
continue
to
drag
along
the
older
one
as
well.
So
you
know
we
recognize.
The
users
often
want
to
stick
with
their
existing
vendor,
just
because
they've
got
existing
storage
and
kind
of
modes
of
operations
with
that,
so
our
goal
is
kind
of
to
support
all
of
them
and
not
be
a
king
maker
and
say
this
is
your
best
solution.
B
One
thing
that
we
could
cover
steve
is,
I
know,
I
know
the
engineers
that
have
written
the
vadp
plugin
for
valero,
so
specifically
on
on
vsphere,
so
how
it
does
vsphere
level
snapshots
orchestrated
by
valero.
So
maybe
we
could
get
one
of
the
engineers
to
come
along
and
talk
about
how
we
do
vsphere
level
snapshots
through
something
kubernetes
level
like
valero.
A
Okay,
yeah,
that's
that
should
be
quite
doable,
but
scott.
If
you
have
a
particular
vendor
that
you're
associated
with
now,
it
might
help
us
if
you
have
contacts
with
that,
where
you
could
invite
them
to
talk
about
whatever
solution
they'd
have
for
kubernetes
backups
on
the
platform
we're
open
to
that.
E
Yeah,
no,
so
I
as
an
integrator,
I
don't
work
with
a
specific.
You
know,
backup
provider.
I
work
with
whatever
the
customer
uses,
but
I
think
more
so
in
the
sense
of
I
mean,
I
think
also
I'm
a
the
you
know.
The
adp
plugin
for
valero
would
be
great
to
get
a
better
understanding
of,
because
there
isn't
much
data
out
there
for
the
valero
plug-in
for
vsphere.
E
In
terms
of
its,
you
know
how
it's
actually
working,
but
more
so
I
think
one
of
the
things
that's
lacking
out
there
today
is
something
in
regards
to
backups
in
kubernetes,
and
it's
not
backing
up
a
virtual
machine.
It's
right!
E
A
Exactly
the
vm
is
certainly
involved,
and
I
don't
want
to
drop
vendor
names
here
in
this
public
meeting
to
plug
anybody.
But
I
know
some
of
these
vendors
who
used
to
have
vm
backup
solutions,
are
adding
to
this
with
kubernetes
backup
solutions
that
go
side
by
side
with
that.
So
that's
kind
of
what
I'm
talking
about
here,
where
what
you
want
to
do
is
back
up
the
persistent
volumes
kind
of
detached
from
the
vms
that
happen
to
randomly
be
the
one
that
they
were
running
on
at
the
moment.
A
You
did
the
backup
but
allow
for
a
mechanism
where
you
can
reconstitute
that
stateful
app,
regardless
of
where
it's
running
at
the
time
of
recovery
and
that's
a
really
useful
tool,
and
some
of
these
vendors
can
still
use
the
snapshotting
features
and
the
application
kiosks,
or
at
least
that's
my
opinion
that
they
had
built
out
for
the
vm
world,
because
some
of
the
kind
of
off-the-shelf
kubernetes
solutions
are
more
like
the
style
of
backup
of
what
you
get.
A
Yeah,
the
other
big
aspect
of
backup
I
used
to
work
in
the
backup
field,
but
catalogs
are
a
big
deal
too.
I
mean
you
know
if
you're
really
doing
this
at
scale.
One
of
the
big
issues
here
is
just
knowing
what
you've
got
and
where
it
is,
it
kind
of
can
seem
to
work
in
a
small
scale
of
yes.
I
developed
this.
I
I
can
back
up
this
database
and
then,
two
years
later,
when
you've
got
thousands
of
them-
and
you
want
to
maybe
selectively-
have
a
deletion
program
that
that
can
be
a
big
issue.
C
Okay,
I
think
it
was,
I
think
we
had
previously
back
in
june.
I
think
we
talked
more
about
that.
I
had
a
conversation
with
robert
hammond
right.
A
He's
part
of
the
storage
business
unit
and
I
think
he
may
be
responsible
for
both
storage
and
the
the
backup
that
would
go
to
the
vadp
appliance.
I
think
his
role
is
that
he
actually
was
talking
about
recruiting
other
vendors
too.
So
I
believe,
he's
been
interested
in
getting
people
from
establish.
A
You
know
all
the
backup
vendors
out
there
to
come
and
work
on
getting
solutions
across
the
board
to
to
enable
backup
of
kubernetes
stateful
apps
to
whatever
form
of
backup
the
user
might
prefer,
but
yeah
he
was.
I
think
we
can
get
him
back
for
a
future
meeting.
That
is
by
the
way
that
that
meeting
got
recorded
scott.
If
you
want
to
go
back
and
look
at
it,
but
I
think
it
was
kind
of
an
intro.
It's
not
going
to
give
you
the
answers.
A
A
A
Okay,
it's
six
minutes
before
the
hour
last
call
for
any
other
comments,
suggestions.
C
I
posted
in
the
chat
another
possible
subject
for
future
is
kind
of.
B
A
Okay,
is
this
managing
zones
for
purposes
of
high
availability,
or
you
know
that
other
reasons.
C
Elsewhere-
okay,
exactly
so,
you
may
want
to
set
up
and
say:
hey
only.
You
know
these
pods
only
work
with
these
zones
so
only
put
them
on
these
nodes
so
that
they
have
storage
available.
A
Okay,
okay,
I'll
I'll,
make
a
note
of
that.
The
other
thing
I
know
that
you're
running
some
on-prem
stuff
and
I've
been
kind
of
playing
around
with
on-prem
myself
and
wondering
if
one
of
the
aspects
of
zones
might
even
be
dedicated
io
devices
that
aren't
available
anywhere.
You
know,
because
I
I've
taken
kind
of
a
wild
ride
on
trying
to
host
home
automation
on
a
kubernetes
home
lab,
and
some
of
my
io
is
unique
to
one
esxi
and
isn't
routable
into
another.
A
But
I
can
imagine
that
in
in
some
of
your
use
cases
you
might
have
some
of
that
going
on
too.
C
Yeah
another
example
would
be
like.
Maybe
you
have
some
gpu
nodes
and
you
want
some
more
clothes
to
run
there,
not
on
the
other.
The.
A
C
A
Okay,
great
idea,
so
we'll
we'll
try
to
find
somebody
to
talk
with
authority
on
those
subjects
too.
Thanks.
A
Okay,
final
last
call:
okay,
I'm
gonna
stop
the
recording.
Then
it
should
get
uploaded
to
youtube
in
a
few
days
and
we'll
see
you
next
month,
bye.
Everybody.