►
From YouTube: Kubernetes UG VMware 20220303
Description
March 3, 2022 meeting of the Kubernetes VMware User Group with discussion of recent updates and patches, changes for Kubernetes version 1.23, and a user survey and discussion of what logging solutions people are using.
A
Hi
welcome
to
the
march
3rd
meeting
of
the
kubernetes
vmware
user
group.
We've
got
a
little
bit
of
a
light
turnout
today,
maybe
because
we
didn't
have
a
speaker
pre-organized,
but
we'll
start
the
meeting
anyway
with,
and
we
did
have
an
announcement
here.
Let
me
share
the
meeting
and
agenda
notes
just
a
minute.
A
So,
just
two
days
ago
a
new
release
of
the
vsphere
csi
storage
plug-in
came
out
and
in
fact
the
docs
just
got
updated
yesterday.
So
there
were
a
few
things
I
wanted
to
call
out.
There's
some
pretty
major
features
for
some
like
snapshotting
is
now
supported
for
block
volumes,
so
that
could
potentially
be
a
big
deep
deal
for
people
who
are
using
backup
tools
like
the
open
source,
valero
or
others,
because
I
think
that
is
a
nice
enablement
feature
for
supporting
these
disaster
recovery
scenarios.
A
A
I
had
proposed
that
we
also
talk
about
the
survey
of
users
members
and
what
they're
using
for
a
logging
solution,
but
due
to
the
right,
like
turnout
today,
maybe
we'll
postpone
this.
The
idea
I
had
here
is
that
often
we
come
up
with
scenarios
where
people
are
expressing
difficulties
troubleshooting
when
you've
got
multiple
layers
of
infrastructure
going
on.
You
know
you've.
You
start
with
an
under
layer
of
a
hypervisor
on
the
vsphere
platform.
A
Often
a
good
tool
for
establishing
causation
is
actually
to
see
you
know
which
layer
had
the
errors
occurring
first,
if
at
all,
and
but
like
I
said
some
of
the
people
who
often
are
vigorous
participants
here,
don't
seem
to
seem
to
have
other
things
to
do
today.
So
I
think
I'll
try
to
roll
that
into
the
meeting
for
next
month.
I
see
we
have
somebody
who
joined
a
little
late.
It's
up
to
you
whether
you
want
to
call
yourself
out.
Oh
okay,
we
we
do
have
people
joining
late.
A
So
you
missed
it,
but
I
just
covered
you
can
refer
to
the
agenda
notes
document,
but
I
just
called
out
that
there
was
a
new
release
of
the
csi
storage
plug-in
and
it
has
some
pretty
major
features.
Just
quick
recap:
the
support
for
snapshots
is
now
there
on
block
volume,
so
that
could
be
a
big
deal
with
backup
and
dr
scenarios.
A
There
are
also
some
known
issues,
one
in
particular.
I
think
at
least
one
person
has
noticed
where
a
partitioning
of
your
vsan
can
cause
some
difficulties
up
at
the
kubernetes
player.
So
I'd
strongly
recommend
that
anybody
look
at
the
release
notes
before
moving
on
to
that
the
docs
seem
very
thorough
and
we're
just
updated.
Yesterday.
A
A
B
And
actually,
the
new
version
comes
with
huge
benefits,
though,
for
those
running
on
kubernetes
when
it
comes
in
to
open
vm
tools,
the
new
or
relatively
new,
like
within
the
last
year,
they
added
app
info
that
comes
out
from
vm
tools
as
a
guest
info
kind
of
thing.
So
you
can
query
from
outside
any
processes
that
run
within
a
vm.
You
can
get
that
as
an
ovf
property
that
gets
updated
every
10
minutes
in
this
version
they
added
that
for
containers
running
either
on
container
d
or
dock
or
get
bubbled
up
to
vsphere.
B
That
can
be
queried
from
a
vsphere,
ui
and
vsphere
api.
So
you
could
actually
tell
now
from
a
vsphere
perspective
which
containers
are
running
on
which
nodes,
which
could
be
helpful
in
different
automation,
scenarios
around
kubernetes
without
needing
to
access
kubernetes
itself
to
understand
where,
like
system
components,
are
running
or
things
like
that.
B
Exactly
I
think,
it's
just
a
good
thing
for
automation
tools
around
things.
I
don't
have
many
use
cases
for
it
necessarily,
but
I
think
getting
that
information
into
vsphere
and
not
requiring
to
have
cube
cuddle
access
and
still
allowing
some
automation,
especially
around
like
maintenance
windows
and
things
like
that,
where
you
can
make
sure
not
to
mess
up
with
things
from
a
vsphere
api
perspective
when
dealing
with
maintenance
could
be
definitely
something
that
will
be
interesting
to
see
how
people
utilize
that
functionality.
A
A
If
the
storage
isn't
thoroughly
integrated
into
vsphere
itself,
and
a
frequent
thing
that
comes
up
during
these
meetings
is
a
situation
where
people
seem
to
have
a
performance
or
a
failure.
Issue:
try
to
troubleshoot
it
get
down
to
a
root
cause
and
if
you
were
to
have
in
place
some
kind
of
a
consolidated
logging
that
would
bring
things
together
through
one
tool
and
accessible.
B
The
reason
I
say,
unfortunately,
is
because
anyone
that
has
managed
elasticsearch
at
scale
at
any
scale
over
time
understands
that
it
is
a
beast
of
a
tool
and
it
is
a
pain
to
manage
and
no
matter
how
many
resources
you
give
elasticsearch.
It
is
never
happy
and
always
once
more,
I
happen
to
think
that
grafana
loki
is
actually
one
of
the
best
tools
out
there
and
I
think
that
getting
because
loki
integrates
into
grafana,
which
is
very
commonly
used
for
observability.
B
A
Yeah
I
happen
to
be
now,
of
course,
I'm
biased
being
a
vmware
employee.
I
can
get
free
licenses
for
my
home
lab,
but
I'm
still
using
log
insight.
A
lot
of
it,
I
think,
is
inertia
where
I
started
using
it
years
and
years
ago
and
it
has
held
up
and
still
works.
But
then
again
I've
got
a
pretty
small
installation.
You
know
three
cluster
nodes
and
it
could
be
an
entirely
different
beast
if
you
get
up
there
to
a
real
data
center.
B
Yeah,
it
definitely
has
issues
at
scale
that
make
it
difficult,
especially
if
you're
running
elastic
on
kubernetes
there's
a
lot
of
intricacies
with
doing
that
itself
and
lots
of
issues
with
the
helm,
charts
that
exist
out
there
and
unless
you're,
going
with,
like
the
operator
based
from
elastic,
which
is
definitely
the
suggested
way,
I
would
say
running
it
on
kubernetes,
it's
not
easy
to
maintain
overtime
in
large
environments,
but
yeah,
I'm
personally
not
a
big
fan
of
elastic,
but
we
see
it
a
lot
out
there.
Just.
A
Because
it's
one
of
them
advice
on
loki
and
poke
around
with
that.
When
I
get
some
spare
time
and
just
take
a
look
at
it,
I'm
curious
and
I
think
I
run
into
people
who
are
looking
for
open
source
solutions
for
particularly
when
they're
smaller
scale
sort
of
bringing
up
an
eval
or
a
learning
exercise
kind
of
cluster.
A
A
And
then,
of
course,
you
know
in
this
arena,
there's
a
lot
of
commercial
things
and
cloud
hosted
and
yeah.
I
think
it's
I've
been
hearing
for
decades
that
some
of
these
logging
tools
become
such
beasts
that
in
something
it's
not
a
one
size
fits
all,
because
some
of
these
that
are
super
capable
could
end
up
being
so
large
that
the
act
of
deploying
them
and
maintaining
them
is
as
big
as
your
the
effort
to
deploy
a
small
kubernetes
cluster.
So
they
might
be
kind
of
overkill
for
some
situations.
B
I
completely
agree
with
that.
We're
seeing
I
mean,
there's
a
reason:
you've
got
hosted
elastic
by
like
seven
different
vendors
out
there.
You
have
logins
like
cloud
from
vmware
you've
gotten
all
of
these
systems,
because
when
you
reach
large
scale
managing
on
your
own,
just
the
cost
of
the
storage
on
its
own
and
making
sure
that
you
actually
have
that
when
it
comes
to
regulations
that
people
have
where
you
actually
have
to
keep
your
data
for
x
amount
of
time.
B
A
B
A
A
A
It's
pretty
wild
yeah.
Well,
you
can
you
know
if
you
get
into
the
vendors
that
sell
logging
solutions
like
splunk,
you
can
just
look
at
the
financials
reported.
You
know
in
in
the
stock
market
tracking
services
to
see
how
much
revenue
they're
pulling
in,
and
you
can
tell
that
yeah
it's
it's
a
pretty
big
deal
and
a
pretty
costly
deal.
A
B
Yeah,
no,
I
it's
not.
I
caught
what
you
said
there
at
the
end.
It's
completely
true,
and
I
actually
think
that
you
know
one
of
the
things
that
people
are
realizing
now
also,
is
that
if
you
keep
your
logs
for
long
enough,
I
just
did
a
project
of
this
by
a
customer
where,
when
they
were
keeping
their
logs
for
a
year
and
a
half,
we
were
able
to
build
very
simple
ai
models
around
those
logs
to
actually
gain
a
lot
of
data.
B
A
And
I
can
see
the
value,
particularly
if
you're
doing
ai
machine
learning
kinds
of
things.
You
definitely
need
a
year,
maybe
two
years
worth
of
data
to
make
that
effective,
because
you
know
they
need
to
have
a
training
set
that
would
go
back
to
establish
whatever
is
a
norm.
That
is
a
meaningful
period
of
time.
A
A
couple
days
doesn't
cut
it,
but
if
you
had
regulatory
and
compliance
issues
that
are
going
to
force
you
to
keep
those
logs
for
5-10
years,
I'm
not
sure
there's
any
value
other
than
checking
the
box
to
say
you
had
it
and
paying
for
keeping
that
part
of
it
on
block
storage.
I
don't
really
know
if
there's
any
solutions
that
support
tiering
to
just
kind
of
roll,
that
off
into
archival
storage
that
maybe
isn't
readily
available.
A
But
you
know
at
the
cost
of
keeping
that
on
high
performing
block
storage
for
10
year
old
log
files
would
seem
to
be
pretty
dubious
to
me.
That's
just
my
gut
feel
I
haven't.
I
haven't
really
been
charged
with
running
a
gigantic
installation
that
would
keep
that
much
data
for
that
long.
But
that's
just
my
gut
reaction.
B
Yeah
I
the
way
that
I
view
it
is
so
many
organizations,
especially
in
the
financial
sphere,
have
to
keep
these
logs
for
such
a
long
period
that
it's
just
a
waste
not
to
utilize
it
so
having
it
in
a
tool,
that's
easily
accessible
via
apis
and
things
like
that
to
actually
be
able
to
run
ai
models
on.
It
is
huge
and
that's
really
where
I
think
decisions
need
to
come
in
of
what
the
actual
extension
points
and
what
the
apis
are
of
a
system
like
this.
B
A
Yeah-
and
this
isn't
just
preemptively
spotting
trends
that
would
clue
you
in
on
impending
failures
or
or
bugs,
but
I
can
see
there's
a
potential
security
tie-in.
If
somebody
was
to
help
people
mine
this,
I
think
very
few
end
users
are
gonna.
This
is
something
that
should
probably
be
done
by
a
vendor
or
project,
because
it's
useful
to
everybody.
A
But
you
know
I
could
see
a
scenario
where
this
is
a
security
tool.
If
you
look
at
things
like
ransomware
attacks,
where
they're
attempting
to
secretly
start
encrypting
things
or
somebody
who
slipped
in
bitcoining
mining
into
your,
you
know,
somewhere
into
your
stack,
you
should
be
able
to
spot
these
with
these
logging
tools
in
the
form
of
unusual
occurrences
in
resource
consumption,
and
if
somebody
was
able
to
deliver
that
kind
of
analysis
based
on
the
big
reservoir
of
logs,
I
think
there's
a
lot
of
people
out
there
who
could
benefit
from
that.
B
Yeah,
I
think
it's
in,
I
think
there
definitely
is.
I
think
that
it
becomes
a
bit
more
difficult
with
those
things
because
usually
they're
not
logging,
anything
right.
So
unless
your
system
is
logging,
specific
things
that
can
show
an
anomaly,
usually
malware
isn't
going
to
be
logging.
Anything
in
your
system.
A
B
Right,
I
think
that
that's
going
to
show
up
more
in
like
a
prometheus
right
than
in
logging,
that'll
be
more
like
metric
gathering
that
you
have,
or
you
have
like
weird
spikes
happening
yeah.
There
are
some
interesting
things
happening
there
as
well
in
the
metrics
world.
I
think
the
biggest
difficulty
on
a
vsphere
perspective
is
that
vsphere,
just
doesn't
have
collection
intervals
or
most
tools
that
collect
metrics
from
vsphere
does
not
have
low
enough
collection
intervals
to
actually
gain
a
lot
of
that
data.
So
like
vrops,
is
five
minutes.
B
A
B
That's
why
what
I
usually
do
in
those
cases
I
use
thanos
actually
with
prometheus,
and
then
you
ship
it
up
to
object,
storage
and
then
the
benefit
of
that
is
it's
basically
long-term
retention.
Prometheus
isn't
good
with
storing
data
over
two
weeks.
The
time
series
database
in
there
is
not
good
at
storing
data
for
a
long
time.
B
Thanos
is
built
to
do
that
and
it
actually
provides
a
full
prometheus
api
and
you
have
thanos
side
cars
that
you
just
put
within
the
pod
of
prometheus
and
it
sends
the
data
to
an
object,
storage,
and
then
you
have
a
centralized
thanos
that
basically
can
query
those
buckets
and
compacts
the
data.
The
benefit
of
that
is
that
you
actually
can
do
things
like
archival
tiers
once
you're
in
object,
storage,
you're
in
let's,
say
aws
you
could
create.
A
B
A
A
A
A
A
Okay,
I'll
take
silence,
then
as
nothing
more
to
discuss
for
this
time,
but
thanks
it.
It
is
a
has
been
a
fascinating
discussion
and
if
anybody
has
ideas
for
a
next
month
or
even
something
where
you
might
want
to
ask
me
to
go,
recruit
a
speaker,
let
me
know
if
you
don't
have
that
idea
now
you
can
drop
it
in
the
slack
channel
for
this
user
group
or
put
it
in
the
agenda.
Note
stock.