►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Hundreds of Clusters Sitting in a Tree with Argo CD - Mike Tougeron, Adobe
What if each developer or team could independently provision and run their own Kubernetes clusters with full api access? Learn how to leverage Argo CD, vcluster and Kubernetes' cluster-api to build dynamic and full-access environments with every pull request. Then see how these independent environments can be stitched together to create a unified view for QE, integration testing or demos. All while managing each environment's costs and compliance from a single view on the host cluster.
A
Thanks
everyone,
it's
actually
hundreds
of
clusters
sitting
in
a
tree
with
a
r
g,
o
c
d.
You
know
just
like
the
the
poem
goes
just
real
quick
who
am
I,
I'm,
Mike,
tujron
I'm,
a
lead,
Cloud
engineer
at
Adobe
I'm
on
the
their
or
ethos
platform.
That's
adobe's
kind
of
bespoke,
bespoke
kubernetes
install
it's
very
similar
to
a
cube
ADM
install
just
with
adobe's
business
Logic
on
top
we're
a
fairly
good
size
installation.
A
It's
around
230
clusters,
22
different
regions,
AWS
Azure,
private
data
center,
18,
000,
compute
nodes,
one
of
the
interesting
stats
I
came
across
was
it
was
1.8
petabytes
of
memory.
That
kind
of
caught
me
off
guards
that
was
kind
of
a
cool
stats.
But
you
can
read
the
rest
on
there.
I
don't
need
to
repeat
it
all,
so
that
was
kind
of
cool,
so
I
thought
it'd
be
good
to
start
off
with
some
of
the
problems
we're
facing.
A
Traditional
Dev
clusters
can
be
expensive
and
if
you
deal
in
the
cloud
cost
controls
are
difficult
to
manage.
I
don't
know
if
you've
ever
tried
to
set
up
quotas
in
AWS,
but
they're
very
easy
for
people
to
just
expand,
sometimes
they're
Auto
expanded,
and
sometimes
when
you
pass
a
quota,
they
just
let
you
go
past
them
or
they
go
ahead
and
bill
you
for
them
and
retroactively,
say
Hey.
You
pass
up
your
quota,
so
it
gets
very
expensive,
very
quick
and
sometimes
they're
kind
of
worthless.
A
A
So
if
you're
working
on
a
custom
operator
like
let's
say
Prometheus
in
that
sort
of
situation,
if
you're
updating
the
crd
and
let's
say
for
some
reason,
it's
not
backwards
compatible,
what's
going
to
happen,
is
you
could
break
other
developers
on
the
same
cluster?
So
those
changes
need
to
be
isolated.
A
Clean
environments
for
integration
testing
can
be
very
difficult
to
do
in
a
micro
service
environment.
Perhaps
you
need
to
spin
up.
Services
are
in
multiple
name,
spaces
test
it
all.
Together,
in
one
thing,
it's
very
difficult
to
do
if
you
just
have
access
to
a
single
namespace
inside
of
a
cluster
so
and
like
I,
said,
multi-tenancy
makes
that
a
whole
heck
of
a
lot
different.
A
So
there's
a
few
tools
that
we're
going
to
discuss
today
that
helps
us
address
these
problems
and
we're
going
to
go
through
those
three
tools
now
so
the
first
is
a
tool
called
v-cluster.
This
allows
you
to
build
virtual
clusters
inside
of
a
single
namespace.
These
virtual
clusters
are
pretty
cool.
They
give
you
full
API
access
with
inside
the
confines
and
the
Scopes
of
that
virtual
cluster.
So
I,
don't
know
if
any
of
you
have
looked
at
these
virtual
clusters
before,
but
you
can
create
namespaces.
You
can
give
our
back
controls.
A
However,
you
want,
you
can
basically
do
whatever
you
want
with
inside
that
confines
as
it
creates
pods.
Those
pods,
though,
are
still
synced
to
the
host
cluster
so
that
they
run
on
the
nodes
inside
of
the
host
cluster.
That
means
security
controls
like
Network
policies,
admission
web
hooks
Opa.
That
kind
of
stuff
are
still
applied.
A
If
you
try
to
create
a
pod
with
a
host
map
or
elevated
permissions,
those
can
still
be
blocked,
so
that
makes
it
so
that
some
developer
Junior
developer,
who
you
don't
want
admitting
your
host
cluster,
can
still
admin
that
virtual
cluster,
but
still
have
the
restricted
access
on
that
development
cluster,
where
we
were
talking
about
cost
before
quarters,
can
be
applied
on
the
host
cluster
that
apply
into
this
virtual
cluster.
So,
as
the
virtual
clusters
are
created,
they're
still
pods
on
the
host
cluster,
so
quotas
can
be
applied.
How
many
pods
are
you
creating?
A
It
also
makes
it
a
lot
easier
and
faster
to
provision
this
resource
than
having
to
wait
for
those
Cloud
providers
to
come
up.
You
get
it
inside
of
a
minute
or
two
instead
of
inside
10
or
15
minutes,
I,
don't
know
who
likes
to
wait,
10
or
15
minutes
for
their
PR
to
start
their
CI
testing.
I
know
I,
don't.
A
The
other
tool
that
leverages
this
is
the
cluster
API
by
the
kubernetes
Sig
cluster
lifecycle
group.
This
is
one
you
probably
haven't
heard
of,
but
it
is
up
and
coming
there's
been
a
lot
of
work
on
it,
you're
going
to
hear
a
lot
more
about
it
this
week
at
kubecon,
but
it
allows
you
to
do
declarative,
API,
provisioning
and
managing
of
kubernetes
clusters.
It's
essentially
a
crd
operator.
You
define
your
cluster
definitions
via
yaml,
manifest
The
Operators
inside
of
the
management
cluster.
Takes
that
yaml,
it's
a
series
of
operators.
A
It
builds
your
kubernetes
clusters,
builds
your
surrounding
infrastructure
based
upon
that,
like
we
were
talking
about
earlier
with
cross-plane
during
the
Keynotes.
You
know,
all
that
surrounding
stuff
is
built
in
a
git,
Ops
style
fashion
works
with
multiple
Cloud
providers,
which
is
really
nice
and,
most
importantly,
it's
extensible
via
custom
providers.
So
it's
not
just
built
for
what
the
cluster
API
working
group
says.
A
If
some
third
party
wants
to
create
providers
for
it,
if
you
want
to
create
your
own,
you
can
and
most
importantly
as
it
relates
to
what
we're
talking
about
today,
there's
a
v-cluster
provider.
That's
going
to
really
relate
you're,
going
to
see
here
in
a
little
bit
how
this
helps
us
create
a
cluster
per
PR,
so
that's
really
cool,
but
basically
the
tldr.
This
is
a
really
cool
tool.
I
started
using
it.
I,
don't
know
three
four
months
ago
and
I
just
found
a
lot
of
really
useful
things
with
it.
A
A
A
The
application
sets
are
really
powerful,
especially
the
pr
generator
that's
possible
with
it
that
allows
us
to
create
applications
per
PR,
that's
generated
inside
of
a
GitHub
or
any
sort
of
git
or
SCM
repo
there's
also
Other
Ones,
based
upon
cluster
lists
that
are
generated
already
inside
of
your
Argo
servers
and
there's
a
bunch
of
other
generators
that
are
there
as
well.
So
this
is
going
to
glue
everything
we're
doing
together.
A
So
why
would
I
want
a
cluster
per
PR?
Well,
we've
already
talked
about
a
lot
of
these
things,
but
I
kind
of
wanted
to
highlight
them
specifically,
as
these
are
the
answers
to
some
of
the
problems
that
we
were
talking
about
earlier,
I'm
not
going
to
Deep
dive
into
them
because,
like
I
said,
we
talked
about
them,
but
those
pieces
that
that
solve
of
microservice
developers
needing
to
have
multiple
namespaces
deployed
for
those
applications
that
can
be
solved
by
this
cluster
because
they
have
cluster
admin,
access
that
crd.
A
If
they're
a
cluster
admin
on
a
cluster,
they
can
install
that
and
not
impact
other
developers.
One
of
the
really
cool
things
that
I
didn't
mention
earlier
is
kubernetes
upgrades.
Let's
say:
you're
up
you're
currently
running
kubernetes
124,
there's
a
deprecation
that
happens
in
kubernetes
125..
You
want
to
test
that
your
code
works
with
kubernetes
125
launch
a
virtual
cluster
as
part
of
your
PR.
That's
running
kubernetes
125
test
your
application
code.
Against
that
you
can
do
that.
A
You
can't
Test
new
features
that
the
older
version
doesn't
support,
but
you
can
test
your
deprecations
to
make
sure
that
your
application
code
will
work
with
that
new
version.
That's
really
cool!
Excuse
me.
We
have
some
applications
that
hammer
the
heck
out
of
the
API
tens
of
thousands
of
Secrets
tens
of
thousands
of
config
Maps,
watches
those
and
can
frequently
bring
down
the
API
server
or
FCD
in
the
dev
environments,
especially
as
multiple
developers
run
those
applications
in
the
same
Dev
cluster.
A
It
also
allows
us
to
isolate
and
create
basically
a
Federation
of
API
servers
so
as
each
PR
can
come
in,
they'll
have
their
own
API
server
and
FCD
server
that
they
run
their
code
against.
So
if
they
have
a
bug
in
the
application,
they
would
bring
down
just
themselves,
not
the
entire
set
of
potentially
hundreds
or
thousands
of
developers.
At
the
size
of
adobe,
so
some
really
powerful
reasons
why
we
would
want
to
Cluster
per
PR.
A
So
what
we're
talking
about
here
is
a
flow
that
looks
something
like
this.
A
PR
is
generated
and
at
that
point
rocd
detects
it.
It
then
creates
an
app
because
we're
running
the
application
set
with
the
pr
generator
it
launches
a
Helm
chart
that
builds
a
cluster,
because
that
Helm
chart
is
using
the
cluster
API
to
build
kubernetes
clusters
so
that
gets
applied
to
kubernetes.
It
builds
it.
It
then
uses
Argo
events
to
trigger
a
workflow
that
workflow
again
inside
of
Argo
takes.
A
It
runs
registers
that
cluster,
when
it's
ready
back
into
Argo
CD
as
a
cluster,
and
at
that
point,
since
it's
now
registered
as
a
cluster
inside
of
Argo
CD,
it
can
sync
any
new
Helm
charts
customize.
Whatever
you
happen
to
be
using
onto
this
new
virtual
cluster
that
had
been
created,
and
now
you
have
this
new
environment
that
has
either
application
code
or
is
bootstrapped
with
whatever
operators
you
want
to
have
on
there.
A
A
A
Put
warning
in
caps
before
I
push
this
I
want
to
show
you
what
it
looks
what's
currently
running
in
Argo,
so
right
now
the
only
thing
I
have
up
here
is
this
cluster
workflows
application
we're
gonna,
don't
worry
about!
What's
in
there
right
now,
we're
going
to
talk
about
what's
in
that
in
just
a
moment,
but
this
is
where
the
meat
or
the
logic
is
happening,
we're
going
to
dive
through
each
part.
A
A
A
A
A
We
checked
back
over
to
Argo
itself
and
we
can
see
now
there's
the
pr
and
we
can
see
that
all
of
a
sudden
now
or
here's
the
cluster
that
it
built-
and
we
can
see
here-
there's
also
two
more
tiles.
It's
synced
in
a
Prometheus
stack.
So
that's
in
there.
Now
it's
synced
into
an
Argo
rollout
stack.
So
that's
in
there
now
and
also
we
have
this
kind
of
environment.
That's
been
bootstrapped
for
this
cluster.
A
A
Okay
and
here's
all
the
pods
that
are
running
inside
of
that
virtual
cluster,
so
it's
kind
of
been
bootstrapped.
I
didn't
have
actual
application
code,
that's
running,
but
you
can
kind
of
get
the
concept
that
PR
has
application
code.
It
does.
The
same
sort
of
thing
applies,
it
moves
on,
does
its
thing,
and
but
the
cluster
has
been
bootstrapped
with
those
generic
resources
as
well
that
that
application
may
need
to
use
so
because
every
application
is
going
to
need
some
sort
of
crd.
It's
going
to
need
monitoring
that
it's
not
manipulating
itself.
A
So
let's
go
back
to
this
and
shoot
I
lost
my
timer,
so
where
am
I
on
time?
Okay,
so
everybody's
got
pencil
and
paper
right.
You
can
write
down.
All
of
this
works,
I
hope
I
have
a
friend
who
does
it
that
way?
That's
how
I
know
he's
paying
attention,
but
don't
worry
all
this
code
is
on
that
repo.
So
if
you
want
to
see
how
this
works,
this
is
the
screen
to
take
a
screenshot
of,
or
a
picture
of
so
I'll.
Give
you
a
few
seconds
to
do
that.
A
A
So
the
pr
generator
application
set-
I,
don't
know
who's
seen
this
or
not,
but
the
top
bit
of
code
here
that
you
see
I,
don't
you
see
my
mouse
around
the
pull
request
area?
Okay,
good?
What
that's
telling
you
there
is
go
to
GitHub.
Look
to
this
repo
for
every
pull
request.
That's
there
generate
an
application
tile
name
it
with
this
name.
A
Then
look
to
this
location
here
for
a
Helm
chart
or
for
whatever
it
is
in
this
case,
it
happens
to
be
a
Helm
chart
could
be
customized
whatever
you
want
with
v
cluster,
ignore
this
control
plane
endpoint.
We
don't
want
Argo
to
constantly
be
syncing
it,
because
cluster
API
will
automatically
populate
that
control.
Plane
endpoint
as
the
Clusters
created,
so
not
much
to
this
one.
But
it's
important
to
point
that
out
how
the
pr
application
set
works.
A
A
If
you
don't
specify
load
balancer,
you
use
a
local,
you
can
use
port
forwarding
to
get
to
it
or
a
local
Ingress
to
get
to
it
or
there's
a
couple
different
options
of
how
you
can
connect
into
this
cluster
security
is
important.
I
like
calling
this
out
on
slides,
even
though
it's
not
there.
He
said
in
you
know
who
can
connect
into
this
cluster
people
scan
ports
everywhere.
You
don't
want
people
connecting
into
this
virtual
cluster
having
admin
access
running
bit
currency
on
your
clusters.
Please
don't
do
that.
A
You
can
Define
what
kubernetes
version
you
want
to
run
and
there's
a
whole
bunch
of
other
Helm
values
you
can
run
on
there.
Basically,
a
v
cluster
applies
a
Helm
chart
as
part
of
its
bootstrapping,
and
so
that's
where
all
those
values
come
from.
You
can
see
the
API
version
up
here
is
a
that's
the
one
for
a
cluster
API.
A
So
did
actually
get
this
into
Argo
CD.
We
saw
it
happen,
but
what's
kind
of
the
more
depth
on
that
flow.
So
what
happens?
Is
you
have
the
Argo
Event
Source?
What
that
does?
Is
that
watches
for
cluster
API
resource
to
be
added
the
Argo
sensor
watches
for
that
picks
up
that
event
and
then
triggers
an
Argo
workflow
that
Argo
workflow
sits
there.
It
pulls
for
that
v-cluster
to
be
fully
created
to
that
provision,
state
that
we
saw
once
that's
there.
It
gets
the
cube
config
once
it
has
the
cube
config.
A
It's
able
to
then
connect
to
that
cluster
and
then
it's
able
to
connect
to
that
cluster
and
to
the
Argo
CD
cluster
or
sorry,
the
Argo
CD
server.
Add
the
cluster
to
that
rocd.
It's
then
able
to
sync
those
resources
like
we
saw,
and
now
that
cluster
has
been
bootstrapped
when
you
close
the
pr
the
reverse
happens.
So
let's
take
a
look
at
that
real,
quick.
We
go
back
to
that
PR
and
let's
merge
it.
A
A
A
I
forgot
that
little
trick
with
PowerPoint
apologies,
but
basically
what
happens
is
again.
An
Argo
event
happens
triggers
a
workflow.
The
workflow
does,
in
the
reverse,
it
gets
connects
to
Argo
CD
runs
Argo
CD
delete,
and
at
that
point
the
cluster
is
removed.
It
does,
you
know,
try
to
remove
the
the
permissions
from
that
V
cluster,
but
the
V
cluster
has
already
gone
away.
So
at
that
point
you
know
it's
just
basically
an
or
true
so
that
it
doesn't
actually
fail
that
command.
The
workflow
completes
successfully.
A
A
A
A
A
Yeah,
so
what
it
does
is
it
passes
in
the
parameter
coming
from
the
resource
that
was
found
to
be
created
in
kubernetes,
passes,
grabs
the
name
of
that
resource
passes
it
in
as
the
first
parameter
to
that
script,
so
the
script
can
take
in
the
cluster
name
that
was
created,
so
it
knows
what
to
act
upon
and
at
that
point
it's
just
running
a
workflow.
So
whatever
you
want
to
do
in
that
workflow
inside
your
container
inside
your
script
that
the
container
runs.
However,
you
want
that
workflow
to
execute.
A
A
The
script
itself,
as
we
mentioned,
is
simple
from
the
perspective
of
it
doesn't
do
very
much.
Does
an
Argo
CD
login,
it
look.
Loops
looks
for
the
cluster
being
to
a
provisioned
state
right
here.
Once
it
has
it.
Cluster
CTL
get
the
cube.
Config
set
up
the
cube.
Config
set
the
right
context:
Argo
CD
cluster
ad
Boom,
the
magic
happens
and
that's
how
it
gets
in
there.
Bootstraps
a
cluster
applies
things
into
it
and
all
of
a
sudden.
A
Now
every
PR
has
a
dynamic,
full-featured
API
server
to
go
along
with
that
PR
and
developers
can
do
whatever
they
want
against
it
and
has
that
full
power
of
a
kubernetes
cluster
without
having
to
go
through
an
Ops
person.
Devops
person
have
to
deal
with
some
other
team
to
update
a
feature
you
know
and
feature
development
becomes
really
easy
in
an
isolated
environment.
A
A
If
you
don't
have
questions
now,
some
come
up
later.
As
you
look
at
the
code,
don't
hesitate
to
reach
out.
I
will
be
taking
questions
in
a
moment.
Some
other
talks
that
we
have
are
up
there.
Friday
Dan
from
code
fresh
and
another
colleague
of
mine,
Joseph
Sandoval,
will
be
diving
into
some
performance
testing
and
some
scaling
stuff
that
we're
doing
as
we
move
along
this
journey
with
stuff.
A
You
want
to
learn
a
little
bit
more
about
the
pr
generator
and
doing
more
about
application
environments
per
PR
Brandon
from
code
fresh
will
be
doing
a
talk
later
this
afternoon
as
well.
So
there's
a
lot
more
that
can
be
done
with
this
sort
of
process,
the
sort
of
philosophy
of
doing
things
on
a
per
PR
basis.
Any
questions
I
saw
lots
of
hands.
I
saw
yours
first.
A
So
the
question
is:
is
when
using
a
virtual
cluster,
do
you
use
Daemon
sets,
or
do
you
Scope
that
down
like
you,
would
in
the
host
cluster?
That's
really
up
to
you.
The
way
V
cluster
works
is
that
it
you
can
sync
in
all
of
the
host
clusters
nodes
or
you
can
only
or
you
could
sync
in
just
the
ones
that
have
pods
running
on
them
in
our
environment,
because
we're
multi-tenancy,
we
only
sync
in
the
nodes
that
have
pods
running
on
them.
So
once
a
pod
is
launched,
it's
assigned
to
a
node.
A
A
A
Okay,
the
idea
behind
this
came
up
from
we
were
talking
about
the
femoral
clusters.
Developers
wanted
a
fresh
environment
to
test
their
application
code.
We
added
previous
business
unit
that
I
was
in
before
I
was
on
the
ethos.
Team
spent
a
lot
of
time
coming
up
with
fresh
environments
to
do
testing.
We
did
you
know
hashes
on
name
spaces
and
that
kind
of
stuff,
and
it
was
a
huge
pain
in
the
ass
I.
It
just
was
expensive
and
a
hassle
to
build
up
full
clusters
that
do
crd
testing
yeah.
A
You
could
do
it
with
kind
locally,
but
a
lot
of
times.
You
need
to
interact
with
cloud
provider
credentials
that
kind
of
stuff.
So
there's
just
a
lot
of
different
factors
that
came
into
this.
A
lot
of
it
was
with
the
cluster
API
building
real
clusters,
because
this
process
of
adding
and
syncing
we're
going
to
be
doing
this
with
real
clusters
on
real
Cloud
infrastructure.
Not
just
virtual
clusters
got
about
a
minute,
less
left,
yeah.
A
That's
the
part,
we're
still
working
out.
That's
the
approach,
we're
going
for
that's
what
we
want
to
do
this
whole
ephemeral.
Cluster
concept
is
still.
We
have
some
teams
that
want
to
deploy
everything.
Some
teams
that
want
to
deploy
just
a
little.
Some
teams
are
still
saying
they
want.
No
I
want
to
Brand
everything
new
VPC,
you
know
new
VMS
and
we're
like.
Do
you
really
need
all
of
that?
A
So
we
haven't
quite
come
to
a
consensus
as
to
what
people
really
need
versus
what
they're
saying
they
want,
but,
yes,
I
I
do
want
to
be
able
to
do
what
you're
describing
and
how
to
find
that
balance.
I
want
to
leave
it
up
to
the
Developers
for
their
spec
and
and
want
to
be
able
to
say.
Okay,
this
project
will
sync
these
ones.
This
project
inside
Argo
will
sync
these
ones
or
give
that
option
within
a
config
inside
of
their
repo
to
say,
run
this
Argo
workflow.
A
That
will
register
them
in
this
way.
With
you
know,
with
these
labels
and
that
label
will
then
sync
that
way,
so
I
will
be
here
all
day.
Hit
me
up
in
the
hallway
hit
me
up
over
lunch.
Hit
me
up
on
Twitter.
My
email
I
am
on
cncf
and
kubernetes
slack.
I
am
open
for
questions.
I
saw
lots
of
other
hands,
so
don't
hesitate
to
hit
me
up
if
you
have
questions.