►
From YouTube: Deep Dive: Long Term Support WG - Tim Pepper, VMware
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: Long Term Support WG - Tim Pepper, VMware
Late in 2018 community discussion of Kubernetes support coalesced into a proposal for a Long Term Support Working Group. "WG LTS" is the name, but LTS should not imply work is towards what you may already believe to be meant by the term. Rather we seek a new, evolved definition of the Kubernetes release/support cadence, an investigation in a time-limited fashion during 2019 of the questions of "To LTS Or Not To LTS" and "What Are We Releasing And Why And How Is It Best Integrated, Validate, And Supported". Discussion spans complex questions of what support is desirable and viable in the minds of users, vendors, and ecosystem code contributors. This presentation will provide an overview of first half of our 2019 survey findings, describe progress towards and future work remaining to achieve a higher level of critical API stability, and outline the shape of possible support enhancement proposals.
https://sched.co/MkJq
A
Welcome
everybody
trust.
You
found
this
room
all
the
way
back
here
and
hopefully
it's
cooling
off
slightly
compared
to
where
it
started.
This
is
the
working
group
long-term
support,
deep
dive
session.
We
didn't
have
an
intro
session
or
something
in
the
otherwise
in
the
maintainer
track.
So
this
is
a
little
bit
deeper
details
kind
of
a
status
update
on
where
we
are,
but
basically
our
working
group
might
kind
of
summarize
with
the.
What
are
we
releasing?
Why?
How
is
it
best
integrated,
validated
and
supported
as
a
broad
question?
A
So
what
is
the
working
group?
First
of
all,
a
brief
background.
Working
groups
are
about
discussion
and
pathfinding
and
crafting
proposals,
as
opposed
to
owning
code,
which
is
what
states
do
and
doing
cross
sig
type
of
work
and
obviously
support
decisions
or
support.
Questions
are
ones
that
then,
would
span
multiple
SIG's
many
stakeholders.
The
question
of
how
we
support
kubernetes
as
an
open-source
community,
is
going
to
span
all
of
the
SIG's
effectively.
A
So
we
we
kind
of
posed
this
basic
question
because
it
kept
coming
up
in
the
release
area
and
do
we
need
a
longer
support
stream,
yes,
or
no,
and
depending
on
the
outcome
of
that,
this
working
group
aims
to
brainstorm
potential
solutions
to
either
those
passive
paths.
If,
yes,
then,
as
a
community,
we
need
to
change
how
we
develop
and
support
communities
if
no,
we
still
can't
deny
the
experience
of
our
users,
which
is
that
they're
having
some
troubles
in
this
regard.
A
So
if
you
want
to
find
more
about
the
working
group,
we
are
in
the
community
repo
with
all
the
other
ones.
You
can
link
there
to
the
mailing
list
to
a
slack
channel
to
our
regular
zoo
meetings
that
we
have
every
other
week
and
there's
a
variety
of
documents
that
we
are
crafting
and
I'll
have
some
additional
links
through
the
slides,
so
that's
kind
of
what
we're
doing,
but
just
to
back
up.
Very
very
briefly,
even
though
this
is
a
deep
dive,
make
sure
we're
level
set.
A
What
is
LTS
so
we
got
the
the
canonical
example
literally,
the
canonical
example
of
when
two
folks
here
in
the
room
and
they're
bright
orange
shirts.
Most
people
who
are
familiar
with
the
phrase
LTFS
know
it
from
an
Ubuntu
perspective,
LTS,
meaning
long
term
support.
But
what
does
that
really
mean
unless
I
mean
about
2:14
for
64?
What
did
what?
What
is
that?
Well,
it's
a
long
term
period
where
you
get
support
an
upgraded
path
and
bug
fixes
and
those
sorts
of
things,
and
it's
a
documented
support
policy.
It's
not
just
something
arbitrary.
A
It
has
both
aspects
of
length
and
type
of
support,
so
an
LTS
might
be
something
that
says
you
get
security
and
bug
fixes
for
two
years,
or
it
might
be
something
that
says
you
get
bug
fixes
for
one
year
plus
another
year
of
critical
security
fixes
there
isn't
exactly
a
thing
that
is
LTS.
It's
a
documented
policy
as
a
project
we
have
to
choose
what
is
the
right
policy
that
we
can
balance
the
what
our
community
can
deliver
with
what
end-users
are
looking
for,
but
to
make
it
explicit
long
as
something
longer
than
a
year.
A
A
Maybe
we
could
just
admit
that
there's
no
value
add
and
doing
something
that
literally
everyone
does,
and
let's
do
it
once
together
with
quality
in
the
upstream,
where
the
subject
matters
existing
can
validate
for
sure
that
as
an
ecosystem,
we
have
the
critical
security
fixes
correctly
patched
again
using
Linux.
As
an
analogy,
the
Linux
kernel,
having
LTS
support
streams
and,
most
recently
dealing
with
the
spectra
in
meltdown
patches
they've,
had
a
huge
struggle
getting
those
patches
with
the
technical
correctness
into
older
support
streams.
A
This
can
be
very,
very
difficult
to
do
hugely
resource
intensive
and,
at
the
end
of
the
day,
there's
not
a
lot
of
value
out
for
a
vendor
doing
that
and
it's
extremely
negative
value
out.
If
you
happen
to
get
it
wrong
because
you
don't
have
the
subject
matter,
experts
who
are
in
the
upstream
developers
get
benefit
if
we
enhance
our
test
support,
which
is
going
to
be
one
of
the
requisites
of
having
something
for
a
longer
term
support.
A
But
what
are
the
risks
can
cost?
So
this
changes
the
human
staffing
equation
on
things
our
test
matrices
grow.
Instead
of
having
the
the
current
three
prior
releases
that
we
test
kubernetes
against,
maybe
there
would
be
more
and
then
you
get
added
technical
complexity.
You
have
to
ensure
your
age
guys
are
stable
over
longer
time,
spans,
which
again
kind
of
goes
back
to
that
test.
Matrix
growth,
but
it's
a
test,
matrix
interoperability
challenge
and
then
you
have
to
validate.
A
So,
as
a
working
group,
we're
trying
to
do
this
with
an
open
mind,
go
about
the
question
of,
should
we
do
something
different
and
to
understand
the
answer
that
is
developers
on
kubernetes.
We
can't
just
answer
that
question
in
a
vacuum,
so
we
put
on
the
survey
last
fall.
We
spend
kind
of
December
in
a
bit
of
January
crafting
a
survey
of
20
plus
questions.
They
can
hindsight.
A
So
we
had
about
600
people
a
little
over
600
people
start
the
survey
and
334
actually
filled
it
out.
It
was
not
the
best
ratio,
but
we
broadcast
the
link
all
over
the
place
and
we
actually
had
a
fairly
high
ratio
of
people
who
got
past
the
first
question.
So
I
think
people
got
the
first
page
load,
maybe
and
we're
like.
Oh
this
isn't
actually
for
me,
so
the
334
we
had
was
actually
a
fairly
large
portion.
A
Once
you
count
the
people
who
went
on
into
the
survey,
it's
a
pretty
good
number
I
think
on
cube.
Atm
community
had
done
a
survey
and
had
maybe
50
yes,
50
hundred
sort
of
range
of
people
responding.
So
at
this
point
this
may
be
one
of
the
largest
survey
responses,
we've
gotten
back
in
terms
of
trying
to
gauge
how
users
are
upgrading
and
moving
their
clusters
forward.
It's
also
important
to
note
that
this
is
during
q1
of
2019
that
people
fill
out
the
survey.
We
close
the
survey
in
April,
so
q1
2019.
A
That
would
that
the
1:13
release
had
just
come
out
so
that'll
show
in
some
of
the
the
next
slides.
So
who
did
we
get
responding?
We
had
quite
a
bit
of
users
of
kubernetes
who
responded
and
then
kind
of
split
evenly
in
the
the
other
half
we
have
operators,
contributors
to
servers,
hosts
and
vendors.
So
a
pretty
good
mix,
I!
Think
if
you,
if
you
look
at
that
300
plus
number
and
count
by
this,
we
had
150
well.
Actually,
if
I
do
40.
A
A
So
hopefully
we
have
some
meaningful
I,
don't
want
to
say,
like
statistically
accurate,
there's,
no
good
way
to
judge
this
when
it's
kind
of
asking
people
their
opinions,
but
we
have
a
lot
of
representation
relative
to
the
size
of
the
community,
the
vendor
and
support
side
community
as
well,
but
also
users
saying
what
they're
looking
for
on
the
consumption
side
and
the
operators
as
well.
So
I
really
apologize
for
folks
who
are
colorblind.
A
This
may
be
slightly
awkward
to
see,
but
this
is
showing
at
the
bottom
releases
that
are
less
than
version
110,
and
then
it
goes
one
not
ten,
eleven,
twelve
thirteen,
so
thirteen
being
the
newest
release
and
for
those
who
aren't
familiar
when
a
release
like
one
thirteen
comes
out,
it
becomes
the
newest
supported
and
then
there
are
trailing
ones.
So
one
thirteen
112-111
would
have
been
in
support
at
that
point.
So
you,
you
can
get
a
sense
of
for
folks
who
are
operating
clusters.
A
They
have
a
portion
that
are
on
not
supported
kubernetes
clusters,
but
one
of
the
things
we
were
curious
about
was
where
the
line
falls
and
for
for
most
of
these
it's
it's
significant.
It's
not
quite
half,
but
maybe
a
third
to
a
half
of
the
things
that
they're
running
include
older
versions.
So
that's
a
sign
that
people
are
either
choosing
not
to
upgrade
for
reasons
or
struggling
to
upgrade
for
reasons.
A
So
we
also
asked
about
when
they
would
like
to
do
upgrades.
So
it's
one
thing
what
you're
running,
but
you
may
be
running
old
stuff
and
prefer
to
run
faster
and
move
faster,
but
not
be
able
to
for
whatever
reason.
So
we
we
asked
about.
How
often
would
you
like
to,
and
how
often
are
you
successfully
able
to
do
minor
upgrades
and
I
want
to
call
out
here
if
you're,
if
you're
used
to
semantic
versioning,
you
might
think
major
minor
patch
are
the
the
three
numbers
and
kubernetes
we
do
that.
A
But
the
one
is
basically
fixed
at
this
point,
so
I
honestly
believe
from
a
developer
and
user
perspective.
What
we
call
a
minor
release,
this
one
dot,
X
that
this
does
amount
to
a
major
release
from
the
user
and
consumers
perspective.
Frankly,
honestly,
so
these
are
the
the
big
disruption
points.
We
we
release
a
new
one,
every
quarter
and
we
we
sort
of
hypothesize
that
we
might
get
a
bimodal
distribution
here
and
I
think
that's
kind
of
what
we've
got.
A
A
Then
patch
upgrades
the
same
question,
so
the
the
dot
Y.
So
if
you're
on
1.13
and
one
dot
13.6
came
out
a
couple
weeks
ago,
we
how
quickly
are
people
consuming
those.
So
some
people
are
happy
and
interested
to
consume
those
basically
immediately
they
as
soon
as
they
come
out.
They
start
trying
to
ingest
them
into
whatever
their
pipeline
is
for
local
validation
into
their
their
production
clusters,
but
other
people
are
doing
things
less
frequently
or
maybe
more
deliberately.
A
Oh,
we
had
a
room
of
probably
50-plus
people
and
we're
not
quite
bickering
back
and
forth,
but
it
was
a
lively
discussion
and
it
could
have
gone
on
for
hours.
I
would
say
so
I'd
like
to
get
more
feedback
from
a
broader
set
of
folks
here,
even
though
it's
a
small
room.
So
in
the
survey
we
also
talked
about
what
question
what
types
of
problems
people
were
having
and
we're
looking
at
kind
of
technical
and
non-technical
things
and
ones
that
are
rated
relatively
high
or
are
there
on
the
screen,
but
so
I'll.
A
Let
you
read
those
the
non-technical
blockers.
I'll
summarize
is
kind
of
the
the
traditional
IT
sorts
of
friction
points
that
amount
to
doing
slower
deployments
in
production.
The
technical
ones
I
will
summarize,
as
this
project
is
immature,
still
we're
working
on
bettering
our
stabilization
and
bringing
more
of
our
api's
to
a
stable
point,
but
there's
a
huge
amount
of
flux
and
change
going
on
all
the
time
and
that
impacts
our
downstream
users.
A
Ultimately,
and
you
see
that
in
the
data
of
things
that
they
they
worried
about,
and
then
we
also
asked
we
tried
to
get
a
little
more
explicitly
what
people
are
looking
for
out
of
the
range
of
possibilities
and
within
the
working
group
and
those
of
us
in
the
support
ecosystem
and
who've
done
things
like
Linux
distro
support
in
our
pasts.
Things
like
that,
we
we
all
have
our
own
opinions
and
biases
on.
What
we
think
is
the
right
way
to
do
this.
A
But
the
question
is:
what
are
the
users
want
and
one
of
the
things
that
rated
really
highly
was
seamless
in-place
upgrade.
We
we've
wondered
whether
we
should
do
upgrades
as
like
an
immutable
infrastructure
type
of
a
thing:
don't
upgrade
a
node
kill
the
node,
replacing
it
with
something
newer
user
didn't
seem
particularly
interested
in
that
today.
Maybe
it's
because
it's
a
new
paradigm
or
maybe
not,
they
want
a
well-documented
release
cycle
and
a
gradual
rollout
and
a
way
to
to
carefully
and
controlled
move
their
cluster
nodes
forward.
A
They
definitely
want
to
see
the
the
REST
API
is
moving
to
stable
v1
and,
at
the
end
of
the
day,
they
want
to
be
able
to
run
their
apps
without
having
to
change
them.
If
a
new
kubernetes
release
comes
out,
they
want
to
continue
to
run
their
apps,
just
as
they
have
before.
They'd
like
it
to
be
transparent
and
I.
Think
we
would
like
that.
Also
we
can
talk
about.
A
A
Boxed
investigation
will
run
a
survey,
will
kind
of
draft
up
some
change
proposals
around
how
we
do
support
as
a
project
try
to
also
improve
test
related
things
and
try
to
help
focus
activity
on
tapi,
solidifying
and
moving
think
more
things
towards
view
unstable
one,
but
at
the
end
of
the
day
the
question
is
like:
should
we
change
the
the
build
and
release
cadence
and
force
people
to
follow
that,
and
would
that
mean
faster
or
slower
upgrades
or
also
there's
been
a
question?
Could
we
do
faster
and
slower?
So
what
does
that
look
like?
A
We
have?
We
have
two
documents
out
there
that
we've
been
drafting.
This
is
the
first
one,
the
short
link
there.
This
is
what
I
call
the
the
faster
and
slower
variant-
and
this
is
a
eye
chart
I,
think,
but
the
the
gist
is
that
you
have
to
sort
of
branches.
What
we
have
today
for
kubernetes,
you
might
think
of
as
a
double
brandy.
In
this
way,
it's
it's
sitting
there,
iterating
and
and
the
person
who's
kind
of
drafted.
This
up
suggested
that
maybe
it
releases
every
month,
so
kubernetes
starts
releasing
faster,
but
we
do
that.
A
For
11
months
spend
a
month,
then,
instead
of
working
on
features
for
that
month
cycle,
we
focused
just
on
stability
and
that
leads
into
forking
off
a
stable
branch
and
that
that
stable
branch
then
gets
something
for
a
longer
support.
This
is
a
model:
that's
existed
in
other
open
source
projects
and
in
other
software
products
as
well
in
in
history
and
out
there
in
the
ecosystem
today.
So
this
is
a
potentially
viable
model.
A
There's
a
big
document
with
all
sorts
of
details
there
and
then
there's
one
that's
kind
of
the
the
one
that
all
say
is
the
slower
variant
again,
the
short
link
there.
But
the
idea
here
is
to
say
that
you
could
actually
do
this
today.
Right
now
we
only
support
upgrades
between
adjacent
versions,
but
if
you
wanted
to
like
say,
we
hypothetically
declared
that
115
was
LTS
and
then
later,
like
a
year
later,
we
made
a
LTS
plus
one
called
1lt
s.
A
Cube
ATM
enables
you
today
to
do
a
sequential
visit
of
all
those
intervening
steps
and
get
to
121.
This
would
be
a
kind
of
ugly
way
to
do
it,
but
this
is.
This
is
functionally
possible.
If
we
were
to
do
something
simple
and
plain,
and
it's
an
idea
just
to
throw
out
there
again
sort
of
open
mind
not
being
prescriptive
zoo,
is
it
a
working
group
and
then
within
those?
Is
the
question
of
in
place
or
not
and
just
to
make
sure
that
we're
on
the
same
page
word
wise?
A
So
if
you've
got
your
notes,
host
nos
and
you're
running,
something
like
one,
not
13.6
for
your
kubernetes
control
plane,
you
upgrade
that
kubernetes
control
fling
software
to
114.
That's
what
will
be
in
place.
There's
a
normal
software
update.
You've
got
a
machine,
that's
running,
you
run
an
update
or
software
that
changes
a
portion
of
the
software
on
that
machine.
If
we
went
with
immutable
infrastructure,
this
one
dot,
13.6
kubernetes
host
combination
once
deployed
we
would
not
want
to
touch
and
mutate
to
any
mutation
would
cause
a
new
one.
A
A
A
C
C
A
Originally,
we
were
thinking
about
splitting
out
separate
surveys
for
the
different
personas
and
we
ended
up
with
just
a
single
one.
So
everybody
answered
the
same
set
of
questions
with
the
first
one
being
an
indicator
on
their
persona,
and
then
we
asked,
if
you're,
trying
to
if
you
you
feel
multiple
personas
match
what
you
do.
If
you
do,
you
pick
the
most
important
one
first,
but
please
fill
it
out
multiple
times
if
you
feel
like
it.
So
we
we
capture
as
much
as
possible.
C
A
We
need
to
double
check
that
we
have
the
it's
really
a
huge
spreadsheet.
It's
like
600
by
50
rows.
We
need
to
make
sure
that
there's
nothing
PII
in
there.
But,
yes,
we
we
intend
to
make
the
fully
anonymized
info
available
for
the
community
and
we
can
build
more
charts
and
stuff
like
that
as
well.
I.
D
So
for
some
club
providers,
who
may
do
some
enhancements
in
the
kubernetes
Cobain's,
sometimes,
for
example,
we
want
to
have
some
actual
features
they
own
through
a
specific
crop
provider
and
terrible
things
happens
if
you
rebase,
you
know,
for
example,
it's
especially
when
the
announcements
accumulates
and
hundreds
of
comments
yeah,
but
I
do
think
that
it's
part
of
the
kubernetes
lifecycle.
How
do
you
think
the
the
LTS
will
resolve
the
case
of
the
case?
Yeah.
A
If
so,
what
one
of
the
things
that
I
think
the
the
steering
committee
for
kubernetes
really
wants
to
ensures
that
we
don't
fracture,
we
need
to
ensure
the
kubernetes
means
something.
If
we
get
too
far
apart,
then
users
will
have
poor
experience
and
kubernetes
doesn't
mean
a
single
thing.
It
becomes
sort
of
a
forest
of
things
and
the
the
user
has
a
much
poorer
experience
towards
that
end.
There
is
conformance
testing
that
assures
that
kubernetes
means
something.
A
If
there's
a
specific
one
you're
talking
about
to
put
you
on
the
spot,
the
the
cloud
providers
should
be
able
to
run
at
their
own
cadence
and
do
so
safely
implement
new
features
and
not
have
to
try
to
wedge
things
into
the
core.
But
again
things
are
maybe
a
little
immature
for
that
just
yet
today,
but
ideally
in
the
future.
That
won't
be
so
much
a
problem.
A
If
we
are
gonna
have
a
well
supportable
longer
span
of
time,
then
we
have
to
have
interfaces
that
are
stable
enough,
that
both
vendors
and
distributors
and
support
entities
can
do
their
value-add
without
having
to
do
these
big,
complicated,
rebase
--is.
The
other
thing
was
sort
of
just
my
generic
open-source
advocacy
hat
on
that
I
would
say,
is
you
should
have
CI
you?
Should
you
shouldn't
just
have
to
rebase?
A
A
You
should
be
able
to
immediately
open
an
issue
or
start
interacting
with
the
open-source
community
around
that,
so
that
we
can
know
that
we
did
that
and
collaborate
on
a
face
if
you
completely
fork
off
to
the
side
and
do
your
own
thing,
you're
gonna
always
be
in
that
type
of
pain.
Oh
can
I,
put
you
on
the
spot,
which
cloud
provider
do
you
have
in
mind,
so
all
admit,
I
work
for
VMware.
We
have
the
same
problem
right,
everybody
does
the
canonical
people.
Do
you
choose
what
you
do
as
well
right,
Red
Hat?
A
D
D
E
D
D
A
Jacob
on
the
back
had
mentioned
test
grid,
so
just
for
folks
who
maybe
aren't
familiar
test
grid
gates.
Do
you
get
this
UI
and
there's
a
conformance
section
and
I'm
gonna,
I'm,
gonna,
admit
I.
Think
I,
don't
like
the
results
here
yeah,
but
so
you
can
see
on
GCE
is
present.
Openstack
is
present.
The
reason
I'm
sad
is
vSphere
is
showing
red
right
now
some
failures,
but
the
more
cloud
providers
we
get
in
here.
E
A
Your
your
description
of
the
the
complexity
of
managing
your
patches,
if
nothing
else,
there's
like
that
mental
cognitive
load
on
the
project,
people
trying
to
understand
all
of
the
different
contexts
in
each
of
these
cloud
providers
that
they
might
be
impacting,
and
each
of
us
can
hardly
understand
our
own
stack,
it
so
complicated,
much
less
all
of
them
in
parallel.
So
this
type
of
signal
is
is
really
huge.
E
Else
have
a
question
tonight:
sorry
just
to
jump
in
the
slide
on
what
the
responses
were
about
upgrade
cadence
I
remember.
There
were
several
questions
on
that
in
the
survey
and
this
says:
when
do
you
upgrade,
and
you
characterize
it
as
when
do
you
want
to
upgrade?
What
is
this
data
coming
from
specifically
so.
A
I
honestly
need
to
double
check
that
it's
been
a
crazy
couple
of
weeks
with
conferences,
and
somebody
else
made
this
slide,
one
of
the
other
folks
in
the
sig
and
when
I
saw
I
had
the
exact
same
question.
So
I'm
not
sure
if
this
was
the
win,
do
you
upgrade
or
when
do
you
want
to
I?
Think
it's
the
win.
Do
you
want
to
okay.
A
We
just
got
this
and
have
made
these
first
couple
of
graphs
and
in
the
I
believe
that
the
next
working
group
meeting
we
will
be
trying
to
present
a
lot
more
will
get
the
the
raw
data
out
there.
So
people
can
crunch
it
and
look
at
it
from
different
directions
and
then
yeah
we.
This
is
something
that
periodically
we
should
do
to
judge,
especially
if
we
end
up
making
a
change
to
the
support
stance
or
if
we
propose
something
and
adapt
it,
then
we
really
need
to
after
a
period
of
time,
redo
this
and
see.
E
A
To
mention
so
a
lot
of
the
sick
meetings
if
you're
familiar
with
the
kubernetes
community,
a
lot
of
them
happen
'morning
specific
time
and
we
have
a
afternoon
meeting
normally.
But
then
we
also
have
one.
That's
like
I,
don't
know
I'm
west
coast
us
also,
but
for
me
it's
middle
of
the
night
and
it
works
out
to
reasonable
for
Europe
and
okay
for
Asia.
So
we're
trying
to
bring
in
more
broader
interactions.
E
Yeah
so
definitely
applaud
that,
but
one
of
the
things
that
Jordan
passed
on
to
me
was
that
there
were
some
other
constraints
around
LTS
for
kubernetes,
based
on
its
dependencies,
the
go
language
some
of
the
packages
that
we
import
and
the
transitive
implications
of
LTS
for
kubernetes.
Can
you
talk
more
about
some
of
that
yeah
I.
A
Would
say
in
a
way
that's
kind
of
how
I
first
came
at
this
issue
on
the
release
team,
when
people
were
saying
they
were
unhappy,
but
at
the
end
of
the
day
there
was
only
so
much
I
could
do
in
my
one
release
cycle
to
worry
about
getting
the
release
out,
and
this
came
up
at
the
I
think
the
end
of
1.10
right
before
we're
gonna
do
the
release.
Somebody
said:
oh,
we
forgot
to
update
the
dependencies
list
for
the
release
notes
so
everybody's
like
well.
A
What
are
our
dependencies
and
we're
searching
like
there's
a
hundred
different
files?
You
got
to
look
at
to
find
what
versions
of
what
things
got
tested,
and
that
was
the
kind
of
the
aha
moment
to
me
like
oh
wait.
We
Tim,
Hakan
and
Brian
grant
back
ed,
Q,
Khan
and
Austin
did
a
talk
about
his
kubernetes,
a
kernel
or
a
distro,
and
that
was
my
aha
moment
seeing
that
list
of
all
of
those
dependencies
and
the
specific
versions
to
say.
Oh
we're
kind
of
a
distro
already
and
I
feel
at
this
point.
B
The
problem
is
because,
like
we
actually
discussed
just
going
as
a
dependency
and
the
problem
is
that
going
back
six
months,
I
believe
it's
a
year
a
year,
okay,
but
it's
a
year
based
on
billing
version.
We
don't
necessarily
upgrade
to
the
newest
billing
version
right
away.
So
it's
often
a
lot
less
than
a
year
for
us,
and
so
then
the
question
is
to
say:
hey
what
does
it
mean
to
have
this
LTS
version
of
kubernetes?
B
C
C
At
this
point,
so
the
real
paradox
that
we
see
is
that
almost
everybody
we
shouldn't
say
almost
every,
but
it's
certainly
a
prevailing
trend
among
customers
and
other
evaluators
is
people
want
the
people
want
to
start
on
current
and
then
have
a
support
window
and
the
next
guy
wants
to
start
on
the
next
quarters.
You
know
and
that's
that's
the
challenge,
long-term
support
draft,
but.
A
For
me,
that's
the
the
beauty
of
the
immutable
architecture
or
the
if
you
take
containers
and
think
of
them
as
immutable
software
infrastructure.
If
you
take
that
one
level
down
the
host
and
treat
treat
the
host
like
that,
you
get
that
choice,
you
deploy
something
and
you
can
keep
it
around
for
a
while
and
then
you,
you
kill
it
off
and
you
manage
you,
don't
have
to
manage
a
whole
set
of
risks.
Risks
in
particular
like
on
this.
A
Where
is
that
picture
one
of
the
the
major
issues
that
the
Jordan
Jordan
Liggett
on
the
huge
ACO
mentioned?
He
brought
up
in
this
picture
on
the
the
left.
Our
most
common
thing
that
we
hear
about
is
you're
starting
to
roll,
something
out,
there's
an
issue,
and
you
have
to
roll
back
partially
like
in
an
RPM
space.
You
you
you've
partially
apply
to
rpm
and
you
need
to
NAR
p.m.
install
it.
A
This
sort
of
stuff
is
really
tricky
when
you
are
mutating
systems
that
aren't
they
don't
have
great
handling
for
that,
especially
in
the
backward
direction
where
the
upgrade
software
is
meant
to
move
you
forward,
and
the
same
goes
for
all
of
the
internal
data
structure.
Even
if
you
you
go
with
just
this,
this
type
of
movement
on
the
right,
you
still
have
the
internal
data
that
needs
to
have
been
version
and
have
translation
mechanisms
to
support
that.
So
it's
not
it's
not
just
a
silver
bullet.
A
Well,
we
are
a
minute
over
time.
I
think
and
I
have
something
I
have
to
run
to
and
four
minutes
to
present
at
again.
So
thank
you
all
for
coming.
I
really
beg
you
all
of.
If
this
is
of
any
interest
to
you,
please
come
get
involved
in
the
sig
share
your
your
ideas.
We
need
more
hands
on
deck,
especially
doing
things
like
enhancing
test
cases
and
getting
better
test
coverage
now.
Thank
you
all.