►
From YouTube: Distribution Team Demo - Aug 24 - vcluster
Description
Distribution team demo on vcluster
A
A
The
overall
purpose
of
this
epic
is
to
take
a
moment
to
think
about
how
we
manage
our
clusters.
What
we've
noticed
a
lot
so
far
is
that,
given
how
rapidly
kubernetes
releases
new
minor
versions,
especially
we're
having
trouble
keeping
up
with
the
process
of
provisioning
clusters
and
installing
dependencies
on
them
and
then
connecting
them
into
CI
and
updating
the
CI
configuration
to
reference
them,
it
tends
to
be
the
longest
part
of
adding
support
for
a
new
version
of
kubernetes
longer
than
just
updating
the
components
that
need
new
API
versions
or
spec
references.
A
That
kind
of
thing,
and
so
we've
been
looking
into
some
options.
For
example,
developers
like
to
use
k3s
or
k3d
or
kind
locally
for
spinning
up
quick
clusters
for
testing.
So
we've
done
some
testing
of
that
in
CI
as
well,
and
then
we've
also
taken
a
look
at
V
cluster,
which
I
believe
Dustin
recommended.
So
thanks
for
that,
so
I'm
going
to
quickly
share
an
update
on
where
we're
at
with
our
at
with
those
and
can
give
you
an
idea
of
what
this
looks
like
right
now.
A
So
this
is
the
Epic
length
here
in
case
you're
curious,
so
we've
done
testing
so
far
with
k3s
kind
and
B
cluster
I'll
start
with
kind.
So
this
has
been
our
long-running
developer.
Local
testing
solution
and
it's
a
great
utility.
The
difficulty
here
when
testing
it
for
CI
is
that
we
need
a
node
that
has
at
least
Docker
and
kind
installed,
as
well
as
some
other
helpful,
tooling.
A
Actually,
I've
looked
at
the
script
in
a
couple
weeks,
but
it's
something
like
this:
we
need
Cube
CTL,
gitlabrina,
Elm
kind
docker.
So
it's
fairly
straightforward
it'll
be
easy
to
make
a
golden
image
of
this
or
use
something
like
ansible
to
provision
nodes,
and
this
worked
okay
with
the
gcp
instance.
It
came
up,
but
then,
of
course,
there's
some
challenges
in
terms
of
infrastructure.
We
need
to
give
that
Helm
installation,
the
external
IP
of
the
node,
so
that's
a
slightly
different
process
and
then
the
bigger
problem
is.
A
This
is
a
kind
of
a
one-off
node.
So
how
many
releases
can
this
node
take
likely?
One
is
how
we'd
probably
provision
it.
So
how
can
we
tell
in
CI
whether
we're
saturated
on
these
nodes
and
we
need
more
spun
out,
there's
no
kind
of
intelligent,
Auto
scaler,
that's
bringing
up
more
of
these
nodes
when
we
need
them,
especially
important,
because
QA
runs
against
a
publicly
accessible
instance,
and
so
it
needs
to
have
that
infrastructure
around
it
to
be
publicly
accessible.
A
The
other
approach,
instead
of
one-off
instances,
was
to
do
it
within
Docker
and
Docker
inside
just
getlab.com
Runners.
This.
Unfortunately,
timed
out,
you
don't
get
to
make
much
in
terms
of
requests
for
the
pods
that
you
get
under
the
hood
with
getlab.com
Runners
and
unfortunately,
gitlab
itself
is
just
a
little
too
heavy
to
run
in
that
kind
of
environment
reliably,
even
if
it
could
there's
no
way
to
keep
that
environment
running
for
the
duration
of
the
CI
pipeline,
because
QA
needs
to
run
against
it.
A
Pause
me
at
any
time
by
the
way,
if
anybody
has
questions,
k3s
is
a
very
similar
story.
K3D
is
very
similar
to
kind.
So
this
is
really
just
the
tooling
under
the
hood,
but
this
actually
went
slightly
better.
The
release
actually
fully
came
up
in
this
pipeline,
so
gitlab
was
able
to
fully
start
and
then
I
wasn't
able
to
get
it
to
do
it
in
the
next
few
tries.
So
that
was
after
10
minutes
of
failing
to
start
a
lot
of
that
duration
probably
comes
from
needing
to
pull
really
heavy
images.
A
B
A
Implement
moving
on
to
V
cluster,
so
this
is
one
that,
like
I,
said,
Dustin
had
recommended
and
it
is
a
free
and
open
source
tool
that
runs
k3s
under
the
hood.
Actually.
So
the
concept
here
is
that
you
have
a
v
cluster
CLI
and
the
way
that
we
would
configure
this
likely
is
that,
instead
of
having
individual,
gke
or
eks
clusters
that
are
created
at
certain
versions,
We
could
create
one
master
cluster,
probably
in
each
provider
still,
but
one,
for
example,
one
master,
gke
cluster
and
just
let
it
float
to
the
newest
version.
A
The
underlying
version
of
that
cluster
isn't
really
significant
to
us.
We
would
use
that
as
a
platform
for
individual
virtual
clusters
managed
by
b-cluster
it's
effectively.
What
happens
is
v-cluster
will
create
a
namespace
in
that
cluster
and
deploy
k3s
to
it,
and
it's
got
some
other
components
as
part
of
it
that
keep
it
isolated
to
that
namespace,
and
so
you
just
tell
V
cluster
what
namespace
name
you
want
and
the
version
of
kubernetes
you
want
it
to
run.
A
This
is
effectively
just
a
stateful
set
in
that
namespace,
and
so
it
comes
up
really
quickly
because
k3s
is
so
small
and
it
actually
provides
really
great
isolation
and
you
can
even
control
how
isolated
it
is.
So
when
I
was
testing
this
in
the
merch
request
effectively,
what
I
did
is
I
did
the
same
process.
We'd
normally
do
for
adding
new
jobs,
but
it's
part
of
our
automation,
tooling,
script
added
one
function
that
will
say
if
there's
no
environment
variable
named
B
cluster
name,
then
I
know
V.
A
Cluster
is
not
configured
so
I'll
skip,
but
if
it
is
configured
which
is
just
an
environment,
variable
in
the
job,
I
will
run
V
cluster
connect
to
the
v-cluster
name
so
effectively.
At
this
point,
we've
already
connected
using
the
gitlab
agent,
the
gitlab
kubernetes
agent,
to
get
connected
to
that
kubernetes
cluster,
and
so
since
I
already
have
a
connection
to
that
cluster.
All
I
have
to
do
is
call
the
cluster
connect
and
it
effectively
just
changes.
A
Your
kubernetes
context
to
point
directly
to
that
virtual
cluster,
so
you're
actually
completely
bypassing
the
underlying
clusters,
API
server
and
going
directly
to
the
one.
That's
provided
in
that
namespace
and
then
we
ensure
that
the
namespace
we
want
exists
in
that
virtual
cluster
too,
and
then
we
set
the
context
to
always
point
to
that.
Namespace,
just
like
we
do
for
the
underlying
cluster
and
the
rest
is
much
the
same.
We
connect
to
the
cluster,
here's
a
view,
cluster
name
so
I
just
name
it
tates126.
A
A
Here
so
set
context,
so
this
is
using
the
gitlab
agent
to
connect
to
the
kubernetes
cluster.
So
you
can
see
we
connected
to
the
gke
125
CI
cluster
and
then
because
on
this
job
I
had
that
environment
variable
with
the
v-cluster
name
configured
it
is,
it
will
actually
connect
to
it.
So
you
can
see
it's
using
v-cluster
kates126
in
that
namespace.
There's
a
load
balancer
endpoint.
A
That
gets
you
directly
to
that
API
server
and
control
plane,
and
then
you
can
see
that
it
switched
to
context
to
that
virtual
cluster,
and
so
now
the
rest
of
the
steps
are
the
same.
We
deploy
and
it
waits
and
restarts
toolbox
and
then
saves
some
environment
variables
and
it's
done
now.
The
reason
specs
is
feeling
right
now
is
because
we
don't
have
server
manager
or
external
DNS
configured
which
we
can
look
into,
but
that
requires
shifting
some
infrastructure
around,
but
very
doable.
A
So
to
give
you
a
live.
Look
here
move
my
zoom
bar
I'll
bump
up
the
size
of
this
here.
So
on
the
top.
Here
we
are
looking
at
the
as
if
we
had
connected
only
to
the
125
cluster.
So
if
I
were
to
list
out
the
name
spaces
here,
you
can
see,
we've
got
our
normal
home
charts
win.
This
is
where
all
our
other
workloads
currently
are
deployed,
and
then
we've
got
the
V
cluster
Kate's
126
namespace.
That
was
what
was
created
when
I
ran
the
command
to
create
the
v-cluster
instance
right
here.
A
A
So
these
are
prefixed
with
the
host
cluster
before
it
now
at
the
bottom,
I
have
used
v-cluster,
connect
or
kubectl
switch
context
to
connect
only
to
that
V
cluster
instance,
so
I'm,
actually
bypassing
the
underlying
hosts
API
server
and
connecting
directly
to
the
one
that's
hosted
by
k3s
in
that
namespace.
So
now
you
can
see
if
I
compare
the
namespaces
they're
different
I,
don't
have
I,
have
Helm
charts
when
that's
just
because
I
picked
the
same
name
as
the
underlying
host,
but
you
can
see,
for
example,
I.
A
Don't
have
that
feed
cluster
case
126
namespace
I,
don't
have
the
gitlab
agent
namespace,
which
makes
sense
because
I
don't
have
an
agent
connected
directly
here
and
so
we're
now
in
a
technically
separate
virtual
cluster,
so
I
can
jump
into
Helen
charts,
win
and
here's
my
release
that
the
CI
environment
created
and
so
everything's
running
as
it
would
normally
I
have
ingresses
picked
up
by
the
nginx
that
I
deployed
here.
This
is
where
the
only
limitation
is
at
the
moment
is
that
cert
manager
can't
see
inside
this
virtual
cluster.
A
A
So
I
see
this
being
a
huge
help
back
to
the
root
problem
that
we're
facing
here,
because
the
main
issue
here
is:
it
takes
a
good
amount
of
time
to
spin
up
those
clusters,
write
the
tooling
around
the
management
for
that
and
install
the
dependencies
on
them
and
then
another
amount
of
time
to
install
the
agent
and
get
that
connected
with
direct
configuration
and
then
also
connect
up
CI.
So,
where
I
see
this
helping
is
that
we
could
have
auto
updating
versions
of
the
underlying
host
cluster
and
use
v-cluster
to
manage
all
those
virtual
ones.
A
So
when
we
need
to
add
support
for
127,
it
would
be
as
simple
as
running
this
command
to
create
a
new
one
at
version
127,
and
then
we
just
replicate
the
CI
setup.
I
do
think,
given
how
repetitive
this
these
changes
are.
I
think
we
can
probably
dry
this
up
a
good
amount.
It
would
be
great
to
even
get
to
the
point
kind
of
like
we
tested
into
K
through
S,
or
it
could
do
a
parallel
Matrix
and
can
only
have
one
definition
of
the
jobs
and
just
test
at
each
V.
A
Cluster
version
specified
here
would
be
really
great
and
that's
really,
where
we're
at
they
even
have
a
cert
manager,
plug-in
I
just
found
that
effectively
helps
you.
V
cluster
will
notice
that
a
Certificate
request
is
available
in
that
namespace
and
it
will
forward
that
request
to
the
underlying
host
cluster
to
ensure
that
that
secret
exists
and
copy
it
back
up
into
that
namespace,
but
there's
probably
multiple
Solutions.
We
could
take
for
this
as
well,
so.
B
A
Mostly
it
they've
got
really
good
docs
I'm
still
getting
familiar
with
reading
all
of
them
and
seeing
everything
that
bcluster
can
do,
but
I
see
this
being
a
big
help
for
our
infrastructure,
which
is
one
of
our
big
limitations
and
struggles.
At
the
moment,
lots
of
manual
work
that
take
up
our
time
from
deliverable
actions.
That
kind
of
thing
so
I
will
stop.
There.
B
A
Yes,
so
in
theory,
it
should
I've,
obviously
only
tested
with
one
release
here,
but
because
the
cluster
is
actually
keeping
everything
isolated
to
that
namespace,
except
copying
the
Pod
definitions
out
to
the
other
underlying
host
to
run
them.
Let's
say
we
go
from
one
to
five
different
releases
that
are
in
a
V
cluster
that
is
still
resulting
in
more
workloads
being
put
onto
the
host
cluster.
Therefore,
the
cluster
Auto
scaler
should
still
scale
up
because
it
notices
that
there
aren't
enough
resources
to
fulfill
all
the
requests
to
run
pods
on
it.
C
That's
cute,
like
I,
didn't
quite
get
the
specific,
like
your
inclination
on
this
one,
but
were
you
looking
at
running
a
persistent
v-cluster
for
each
version
of
kubernetes
or
my
idea
would
be
okay,
since
we
have
that
great
flexibility
and
it's
not
taking
much
time
to
provision
one.
We
just
provision
them
on
the
Fly
for
each
particular
pipeline.
C
A
I
think
we
probably
could,
because
the
like,
all
it's
really
running,
is
core
DNS
and
it's
controller,
which
is
k3s
under
the
hood.
So
let
me
see
it's
requesting
200
megabytes
of
CPU
and
256
of
memory,
so
it
actually
is
really
small.
We
probably
could
provision
per
release,
which
I
think
would
be
even
better,
because
then
we're
really
putting
each
release
in
an
isolated
area
yeah,
because
that
would
address
isolation
and
the
issue
of
spinning
up
clusters
and
managing
them.
So
I
think
that's
probably
very
very
likely.
C
Yeah
yeah,
because
then
we
can
actually
a
if
we
need
to
introduce
a
new
cluster
in
the
Mr.
We
just
add
another
iteration
like
into
the
Matrix.
We
just
had
another
127
or
128,
and
then
the
jobs
are
happening
only
for
that
particular
pipeline
until
it's
merged
and
then
yeah,
it
would
be,
I
see
it
if
we
can
make
this
fly.
That
would
be
a
huge
simplification
of
our
maintenance
for
the
CI
cluster
for
sure.
A
A
C
I
think
I
saw
somewhere
that
it
actually
under
the
hood
prefixes
the
names
of
the
pawns
or
like
when
you
deploy
it
or
was
it
the
namespace
that
it
was
prefixed,
because
we
do
have
one
issue
of
you
know
the
limit
of
63
characters.
So
I'm
kind
of
wondering
about
that.
Are
we
gonna
run
into
this?
Because
there
is
more
prefixes
now.
A
Yeah
we'd
have
to
see
how
that
looks
in
practice.
I
think
it
looks
like
it
kind
of
appends
its
own
hash
to
it,
but
because,
obviously
these
These
are
truncated
and
it
gave
no
problem,
but
we
would
want
to
make
sure
that
releases
wouldn't
like
the
names
of
the
pods
wouldn't
talk
over
each
other
yeah
I
can
test
that
I
could
do.
I
could
do
a
no
approach
in
this
Mr
where,
instead
of
all
deploying
to
the
same
V,
cluster
I
can
make
a
new
V
cluster
for
each
one
and
I'll.
A
D
B
D
And
there's
like
a
it's
guaranteed
to
work
with
like
a
version
ahead
and
a
version
behind,
so
that
would
guarantee
three
versions,
but
it
usually
works
with
even
more
than
that
right.
So
we'd
have
like
a
guarantee
of
three,
but
it's
possible
that
we'd
run
into
a
case
where
okay,
we've
hit
a
version
that
now
doesn't
work
on
this
node
version.
Yeah.
C
D
B
D
Bring
it
up,
it
means
we
should
still
when
we're
redoing
this
CI.
We
should
still
anticipate
that
additional,
like
actual
clusters,
would
need
to
be
attached
just
so
we
don't
like
fully
focus
on.
We
only
have
one
one
cluster
ever
just
to
keep
it
flexible,
but
yeah
I.
Imagine
it
I,
don't
know
it's
looking
really
good.
A
I
would
love
to
ask
them
how
that
is
translated
on
the
openshift
point
to
this
can
run
rootless
mode,
which
is
great,
so
you
can
run
on
an
open
shift.
That
said,
I
don't
know
if
it
helps
us
much
because
I
think
our
purpose
of
deploying
to
openshift
is
that
we're
interacting
with
openshifts
implementation
of
a
cluster,
whereas
this
is
k3s
under
the
hood
yeah.
That's.
C
A
A
A
A
A
This
is
the
the
the
similar
topic
too.
By
the
way,
these
are
the
sync
resources,
so
there's
a
component
with
NV
cluster
that
runs
next
to
k3s
that
copies
objects
into
the
underlying
cluster.
Most
of
these
are
disabled
by
default,
but,
for
example,
secret
services.
So
it's
actually
configured
by
default
that
you
can
talk
to
resources
in
your
host
cluster
as
if
it
was
a
separate
cluster,
because,
theoretically
it
is,
you
can
actually
isolate
that.
So
they
don't
have
permission
to
speak
to
each
other.
A
But
we
might
end
up
using
something
like
that,
for
example,
for
like
a
shared
external
DNS
instance,
so
that
they
can
talk
to
each
other
same
goes
for
cert
manager,
so
something
we'll
look
into.
But,
for
example,
we
could
enable
its
ability
to
copy
Ingress
definitions
back
to
the
host
cluster,
so
that
cert
manager
and
external
DNS
would
still
see
those
requests
coming
in
and
ensure
certificates
and
DNS
records
are
provisioned.