►
From YouTube: How Adobe Planned For Scale With Argo CD, Cluster API, And VCluster - Joseph Sandoval & Dan Garfield
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
How Adobe Planned For Scale With Argo CD, Cluster API, And VCluster - Joseph Sandoval, Adobe & Dan Garfield, Codefresh
Speakers: Dan Garfield, Joseph Sandoval
Adobe has thousands of developers working on over 120k namespaces handling billions of packets per second. When taking the decision to standardize on Argo CD we needed more than a standard implementation. We needed a plan for scale! Argo CD is incredibly powerful, both for its ease of use and for its scalability to manage applications across many target environments. At Adobe, Argo CD started at the grassroots and has grown into something more substantial, the time came to take a more strategic approach to make sure security, scalability, and management are done right. Leveraging Cluster API, VCluster, and a holistic approach to scaling we’ll cover performance benchmarks to help you plan out when you need to divide up instances, as well as security, access, and other considerations when planning for growth. We’ll also demonstrate the right techniques for RBAC, SSO, and other tools to help you scale more effectively. If your organization is starting to grow with Argo CD, you don’t want to miss this talk. There’s more to scaling Argo CD than deploying HA.
A
B
A
B
I'm
Joseph
Sandoval
I
am
with
Adobe,
which
is
just
in
the
title.
I
am
now
a
product
manager
for
in
our
Dev
experience,
Group,
which
really
means
that
I'm
still
working
with
the
underlying
kubernetes
infrastructure
and
then
in
this
alternate
life.
I
get
to
hang
out
with
kubernetes
Sig
release
team
been
part
of
like
five
release.
Cycles
Branch
release
manager
associate
still
trying
to
retain
it,
because
that's
one
of
the
things
I
highly
value
yeah.
A
And
I'm
Dan
Garfield
I
am
the
co-founder
and
chief
open
source
officer
of
code.
Fresh
and
I
also
feel
like
I
leave
a
dual
life
because
I'm
also
an
Argo,
maintainer
and
then
I
helped
start
the
get
Ops
working
group
and
open
git
Ops
as
the
get
off
standards
that
are
cncf
project.
That
I've
been
very
pleased
to
see
a
lot
of
pickup
on
at
kubecon
in
lots
of
people's
slides.
So
we're
going
to
talk
about
Adobe
and
how
you're
scaling.
B
B
That
adobe
has-
and
so,
if
you
hit
up
any
of
the
talks
this
week,
You
probably
there's
a
couple.
People
here
who
have
been
speaking
about
what's
been
happening
there,
but
you
may
have
heard
the
name
ethos.
But
what
ethos
is
it's?
It's
the
cloud
native
platform,
if
you're
going
to
do
anything
there,
that's
the
spot,
it's
going
to
happen
at
and
this
platform
has
existed
for
about
five
years,
and
so
a
lot
of
change
has
happened.
A
lot
of
developments,
the
interesting
things
about.
A
B
Compute
nodes,
1.8
petabyte
of
memory
and
about
500k
thousands
to
be
used
along
with
that
project
is
growing.
I
think
the
count
was
2.3
million
and
along
with
it,
we
were
a
company
of
about
26
000
and
you
can
imagine
a
lot
are
developers.
So
the
great
thing
is
we're
scaling
changes
happening
there
and
it's
exciting
to
kind
of
see
and
we're
pushing
things
as
far
as
it
goes.
B
But
with
that,
though,
that
also
presents
some
challenges
for
us,
because
a
lot
of
us
know
who
are
running
platforms
and
when
I
refer
to
platform,
because
we
have
quite
a
few
teams
who
are
doing
a
lot
of
different
things
there.
The
team
I'm
on
is
really
foundational
and
we
really
care
about
like
providing
that
Bedrock,
that
the
teams
that
are
building
like
the
developer
experience
or
the
pave
path
or
the
teams
that
are
like
these
large-scale
dedicated
clusters
that
they
have
the
space
to
be
able
to
do
the
things
they
need
to
do
so.
B
You
know
we
get
a
lot
of
requests
and
some
of
them
are
hey.
We've
got
an
environment
that
we
want
to
be
able
to
test
our
changes.
We
want
femoral
clusters,
we
want
these
things
to
spin
up
on
demand
and
a
lot
of
us
from
our
side
of
it
are
just
like.
That's
that's
challenging.
How
do
we
kind
of
do
these?
Because
we
have
some
pressures.
B
B
Further,
as
well
as
like
just
getting
out
of
the
way
like
really
just
enabling
teams
to
sometimes
want
to
go
fast,
they
want
to
be
able
to
like
be
on
current
release
versions.
Other
teams
want
to
be
more
guarded
and
we're
managing
this
all
the
while,
while
we
have
Dynamic
times
of
years
where
things
have
to
scale
or
we
have
holiday
periods,
and
so
we
really
have
to
create
this
layer-
and
this
is
where
the
challenge
comes
in
when
I'm
the
team
that
I'm
working
with
is
that?
B
How
do
we
create
that
separation
or
that
sensible
layer?
Because
we
have
challenges
as
well
we're
trying
to
keep
the
fleet
current
trying
to
keep
that
Fleet
consistent?
We
have
components
and
all
these
things
that
make
up
that
underlying
platform
that
really
comprise
it.
It's
not
just
kubernetes.
We
have
tooling
that's
around
security.
B
B
I
think
we're
good
all
right
thanks
for
everybody's
patience
with
this,
as
I
was
saying,
we've
been
around
for
five
years
and
it
was
very
reflective
of
that
start.
As
I
was
mentioning,
they
started
off
with
mesosphere,
which
then
evolved
into
kubernetes,
and
so
a
lot
of
the
way
that
they
built
and
deployed
a
message
for
influenced
what
happened
afterwards,
meaning
they
had
a
very
tightly
coupled
get
repo
that
it
mirrored
the
structure
and
the
org
at
that
time
and
how
it
worked
was
it
was
barely
tightly
coupled
that
was
great.
It
scaled
us
up.
B
This
is
when
things
started
to
get
a
little
bit
more
impedance,
and
this
is
where
we
had
to
start
really
thinking
about
where
we
were
headed
and
that's
because
because
of
that
tightly
coupled
nature
of
how
we
were
building
and
deploying
releases
that
all
changes
needed
to
be
reviewed
by
an
all-knowing
maintainer.
Even
though
you
may
have
this
service.
B
That's
way
over
here
that
you
know
it
very
well,
but
because
it
touched
the
core,
we
needed
these
individuals
to
be
able
to
make
sure
that
it
was
not
going
to
cause
any
implosion
or
any
friction
or
cause
or
release
to
you
know
have
a
problem
so
that
system
had
really
served
its
purpose.
But
now
it
was
time
to
revolve
so
that
we
had
this
idea
and
I
think
we
were
really
kind
of
looking
at
that
time
and
it
was
still
early
in
and
I
think.
B
A
few
of
us
were
looking
at
the
Argo
community
and
we
saw
some
areas
where,
like
hey,
there's
some
challenges
there,
but
it
looks
interesting,
but
we
thought
maybe
we
could
iterate
on
what
we
had
at
that
time.
Maybe
that's
a
fast
way
to
kind
of
like
unblock
ourselves
in
a
sense
kind
of
like
decouple
releases,
and
that
was
our
next
attempt
and
I
think
Dan.
This
is
kind
of
about
the
time
I
met.
You
I
think
I
was
talking
about
this,
where
we
overlaid
Argo,
workflows
and
really
to
do
that.
B
We
really
had
to
create
a
workflow
of
workflows,
which
then
made
things,
maybe
a
little
bit
more
complex,
not
as
like
easy,
but
it
did
unblock
the
trains
of
releases
to
individual
clusters.
So
that's
a
good
thing,
but
the
challenge
of
maintainership
was
still
there.
The
a
lot
of
the
features
that
we
had
to
start
thinking
about
was
kind
of
like
well
wait
a
minute.
Are
we
kind
of
re-implementing
something
that's
out
there,
which
is
Argo
CD?
But
you
know
we
went
with
it.
It
was
kind
of
a
lift
and
shift
approach.
B
A
My
mic,
okay,
so
were
you
very
used
to
using
CI
CD
as
devops
teams?
We've
been
using
it
a
long
time?
We
build
a
pipeline
like
how
many
of
us
have
built
those
pipelines
literally
10
000
times,
like
I'm,
going
to
run
a
build
I'm
gonna
run
some
tests
I'm
going
to
deploy
something
I'm
going
to
run
some
additional
tests.
Maybe
I
build
some
logic
to
do
a
rollback
if
I'm
getting
fancy
and
for
the
for
the
use
case
that
you
needed
to
support.
You
know
which
you
you
clarified,
so
many
diverse
use
cases.
A
You
needed
a
really
complex
workflow
to
support
that.
So
you
had
a
workflow
of
workflows,
so
it
was
intelligently
designed,
but
that
also
meant
that
at
the
end
of
the
day
it
was
very
complex
and
we
all
know
that
in
CI
CD,
it's
very
I
mean
it's
an
imperative
operation
and
so
creating
item
potency
inside
of
a
CI
CD
pipeline
is
really
difficult.
A
I
said
deployed:
did
it
work,
do
I
need
to
add
a
retry
step
and
you're
going
to
do
doing
retries
and
these
kinds
of
things
and
it's
not
event
driven
and
it's
just
because
implementing
item
potency
is
really
hard.
So
this
approach
that
you
had
built,
you
know
it
got
you
from
point
A
to
point
B,
but
you
were
also
looking
at
point
x
down
the
line
and
thinking
you
know.
A
How
are
we
going
to
scale
this
and
then
you
know,
we've
got
Mike
tudoron
on
the
team
who
can
manage
this,
but
we
don't
have.
You
know
15
or
100
of
him
so
how's
that
going
to
scale
up,
and
so
this
is
when
we
started
talking
about
git
Ops
and
I
tried
so
hard
not
to
make
a
face
because
I
was
like
I
saw
you.
This
is
cool.
This
is
great,
maybe
some
things
to
think
about.
So
you
know
this
is
the
approach
was
well
we're
doing.
Ci
CD,
let's
just
extend
it
right.
A
A
A
There
are
four
principles
that
go
into
git
Ops
and
they
this
this
involves
how
we
manage
State,
our
our
actual
State
and
our
desired
State
and
I'm
not
going
to
go
through
all
those
principles
right
now,
but
you
should
read
them
and
potentially
memorize
them,
because,
basically,
what
we're
trying
to
do
is
keep
track
of
our
actual
State
and
make
sure
that
there's
no
drift
drift
is
when
something
changes
in
our
state
that
was
not
defined
in
our
source
of
Truth.
It
was
not
defined
in
git
and
I.
A
I
talked
to
a
lot
of
people
who
you
know.
You're,
like
hey
man,
I've
been
using
terraform
for
five
years,
I
check
it
into
git
I
apply
my
plan
I
feel
like
I'm.
Doing
get
up
smells
like
it
smells
smells
kind
of
like
Get
Ops
and
it's
and
you're
you're
at
least
halfway.
There
I
actually
talked
with
somebody
recently
who
told
me
that
they
they
went
to
actually
check
all
their
terraform
and
they
found
that
over
half
of
their
terraform
plans
that
were
old
had
drifted
and
they
actually
couldn't
easily
reconcile.
A
So
that
means
that
things
were
getting
changed
in
production.
That
they
weren't
aware
of
and
that's
shockingly
common,
so
the
get
off
style,
is
that
you
have
a
source
of
truth.
That
source
of
Truth
is
get
in.
The
years
have
passed
when
we're
doing
CI
CD
pipelines
a
lot
of
times.
Our
source
of
Truth
is
sort
of
inferred
by
the
operation.
That's
happening
in
a
pipeline,
but
what
happens
when
a
step
fails?
What
happens
when
a
web
hook
fails?
A
What
happens
when
the
state
has
changed
and
you
can't
detect
it
so
you're
trying
to
push
changes
onto
something?
That's
not
what
you
expect.
All
of
these
situations
are
the
reason
that
get
Ops
was
created
in
the
first
place.
So
in
your
case,
one
of
the
things
that
we
wanted
to
do
was
handle
pull
request
generation.
That's.
A
B
A
A
Yeah,
when
you
have
an
operator-
and
you
go
to
upgrade
it
It's
tricky
to
test
right
and,
in
your
case,
you're,
supporting
not
only
multiple
clouds
but
multiple
kubernetes
versions.
So
having
that
all
operate
together
in
a
git,
Ops
fashion
would
be
very
valuable.
It
is
yeah,
so
a
couple
of
key
tools
for
this
yeah.
B
So
I
think
the
is
in
the
talk
of
course,
I
name
dropped
all
the
names
that
I
love
here,
but
I'll
start
with
V
cluster
I.
Think
this
one
came
on
my
radar
like
last
year
at
La,
kubecon
Lucas
gave
this
great
talk
and
I
was
just
like
wow.
This
is
kind
of
neat
because,
all
of
a
sudden
now
we
can
spin
up
a
virtual
cluster
that
runs
in
an
a
space
as
far
as
our
developers
they
feel
like
they
have
full
API
access.
B
They
can
do
whatever
they
need
to
do,
but
yet
we
can
make
sure
that
the
host
security
is
not
is
enforced.
It's
really
easy
for
Rapid
provisioning,
so
it
was
just
like
a
natural
fit
for
that.
You
know:
hey
I,
want
this
quick
ephemeral
thing
I
want
to
test
this
thing
and
we
want
to
be
able
to
just
let
people
kind
of
test,
as
often
as
they
want,
without
breaking
each
other
yeah.
A
Now
those
V
clusters,
in
this
case
we're
going
to
be
defining
them
using
cluster
API
if
you're
not
familiar
with
cluster
API
cluster
API
is
an
operator
that
allows
us
to
take
manifests
describing
v-cluster
or
other
other
cluster
resources,
and
cluster
API
will
make
sure
that
those
are
spun
up
and
reconciled
so
because
all
of
these
things
are
essentially
just
kubernetes
manifest
they're,
just
kubernetes
resources.
There
is
a
tool
that
is
designed
perfectly
well
for
synchronizing
those
and
it's
called
Argo
CD
hands
who's,
familiar
with
Argo
CD,
okay,
most
people.
A
So
the
get
Ops
thing
people
were
like
I,
don't
know
what
that
is,
but
Argo
CD
everybody's
got
it.
So,
within
the
Argos
project,
there
are
four
main
tools:
there's
Argo
CD,
which
all
of
you
are
familiar
with,
which
is
great.
At
reconciling
resources,
you
define
a
get
source
of
Truth,
you
define
a
destination
and
the
reconciliation
engine
takes
care
of
the
rest,
there's
Argo
workflows,
which
is
a
general
purpose
workflow
engine
for
kubernetes,
and
allows
you
to
run
any
kind
of
workflow.
A
Every
step
operates
in
its
own
pod,
it's
very
popular
with
data
pipelines
and
etls.
But
it's
also
incredibly
popular
for
doing
workflows,
scripted
events
that
need
to
happen
and
then
there's
Argo
events,
which
is
for
Eventing
and
triggering
these
things
and
there's
Argo
rollouts
for
doing
Progressive
delivery.
B
B
I,
think
you
know
it
was
always
been
a
cloud
native
experience
for
developers,
but
as
operators,
we
always
kind
of
like
seeing
the
trail
behind
and
having
that
and
then
seeing
the
maintainer
adoption
over
the
last
year
was
the
confidence
buildup
for
me,
anytime,
I
see
these
projects
like
reach
that
inflection
point
it's
it's
a
great
time
to
jump
on
board
oops
go
back,
so
this
was
kind
of
where
we
landed
at.
This
is
the
architecture
that
right
slide:
yeah,
you're,
good,
okay,
sorry!
This
is
a
architecture
that
we
landed
at.
B
Environment
was
the
kind
of
like
use
case
where
a
developer
opens
a
PR
and
he
wants
to
test
the
operator
of
his
or
application
that
we
need
to
get
full
access,
and
this
application
is
where
the
magic
of
kind
of
building
something
is
or
the
application
said
it's
there
to
the
left
upper
left
is
where
all
the
magic
kind
of
happens
at,
and
so
we
use
this
PR
generator
to
generate
like
a
b
cluster
and
Nick's
example,
with
the
support
of
cluster
API
and
the
workflow
is
pretty
basic.
It's
really
just
you
know.
B
The
Event
Source
is
looking
for
this
new
cluster
API
resource.
The
Argo
sensor
picks
up
the
event
and
sure
goes
a
new
order,
workflow
and
then
waits
for
the
new
clustered
b-cluster
to
be
fully
created
and
gets
the
cube
config
and
all
the
other
things
that
need
to
be
in
that
cluster
into
Argo,
City
I
think
I
may
be
in
the
wrong
spot.
I
think
yeah,
so
Argo
City
from
there
really
sinks
generates
all
the
resources
to
it.
You
know
whether
it's
the
Prometheus
operator,
an
English
controller.
B
You
know
all
the
things
that
make
it
that
unique
property
and
then
all
of
a
sudden,
you're
bootstrapped-
you
got
this
new
v-cluster.
So
for
us
that
was
like
the
really
inflection
point,
and
the
great
thing
about
this
approach
is
that
it
could
also
be
leveraged
into
regular
clusters
as
well,
which
we
also
wanted
to
do
so.
This
was
kind
of
like
that.
Stepping
stone,
I,
don't
know.
If
you
ate
Dan,
you
probably
run
something
similar
yeah.
A
This
this,
this
approach
is
just
so
impressive
and
strong,
because
it
means
that
you
can
test
those
operators
against
multiple
kubernetes
versions,
and
you
can
do
them
in
self-contained
pull
request
environments
that
are
generated
via
application
sets,
but
a
lot
of
people
are
familiar
with
application
sets
as
a
way
to
generate
applications
in
Argo
CD,
but
within
application
sets.
We
have
this
concept
of
generators
and
generators
include
things
like
a
pull
request.
So
all
your
pull
requests
can
generate
applications.
A
All
of
your
git
repos
can
generate
applications,
a
folder
structure,
a
list
any
of
these
kinds
of
things,
and
they
can
be
used
in
combination
with
one
with
one
another
in
what's
called
a
matrix
generator.
So
this
this
environment
also
allows
you
to
save
a
lot
of
money
because
running
a
v.
Cluster
is
ultimately
incredibly
cheap,
considered
compared
to
running
an
entire
cluster.
There's
a
great
preview,
a
prequel
to
this
talk.
If.
B
You're,
yes,
there
is
yeah,
there
was
done
the
other
day
by
Mike
toujon
who's
in
the
audience.
If
you
really
want
to
get
into
a
deeper
dive,
it's
not
posted
yet
I
expect
those
talks
to
come
up
here
in
the
next
few
weeks,
but
it's
hundreds
of
clusters
sitting
in
a
tree
with
Argo
CD,
it's
really
kind
of
like
a
lot
more
in
detail.
You
can
kind
of
find
the
patterns
that
you
may
be
able
to
leverage
yourself.
B
I
think
it
also
has
a
repo
that
you
can
pull
from
as
well
to
see
how
all
this
magic
happened.
But
this
talk.
Yes,
it
is
about
some
of
the
things
that
are
in
my
title,
but
the
key
thing
that
I
always
have
my
eye
on
is
scale
I
have
an
existing
Fleet
I
know
with
how
big
this
Fleet
is
going
to
be
and
where
it's
going
in
one
to
three
years
and
I'm,
trying
to
figure
out
all
right.
We
built
this
thing:
we've
had
scaled
things
in
the
past.
B
This
is
where
I'm
gonna
set
you
up
to
hit
it
out
of
the
park.
I
came
again,
and
this
was
a
true.
This
is
seriously
I
was
kind
of
like
what
are
the
skill
points
like?
Does
anybody
have
any
bass
lines
out
there
that
I
can
pull
from,
because
I
was
looking
out
there,
trying
to
figure
out
like
how
do
I
communicate
to
the
team,
our
plan
of
scaling
and
so
I
think
that's
when
you
just
you're
like
hey.
Let
me
let
me
look
at
this.
What
how
can
we
scale.
A
This
yeah,
who
who's,
read
the
official
Benchmark
report
from
Argo.
That
tells
you
when
you
need
to
consider
scale
components.
Nobody,
oh
because
it
doesn't
exist
right
so
that
that
was
a
call
to
action,
because
hopefully
I'll
write
one
coming
out
of
this.
Henrik
raised
his
hand
because
he's
like
he's
like
dude,
we,
like
Bill,
wrote
the
book
on
scale.
He.
B
A
He
knows
he
knows
what
he's
talking
about,
but
so
scaling
with
Argo
CD
and
with
kid
Ops.
The
approach
that
adobe
is
taking
takes
advantage
of
a
lot
of
the
structure
that
git
Ops,
provides
and
directory
structure
and
pull
request,
generators
with
for
Margo
CD
and
those
kinds
of
things.
So
it
has.
It
has
really
great
Primitives
for
scale,
but
when
is
Argo
CD
going
to
run
into
Performance
challenges?
A
What
are
the
security
considerations
that
we
need
to
make
and
then
are
there
any
organization
structure,
considerations
that
we
also
need
to
take
into
consideration?
There's
a
Blog
that
I
put
together
about
this
called
scaling,
Argo
CD
securely
in
2022.
This
is
a
Blog
that
I
try
to
keep
Evergreen
with
what's
happening
in
the
community
and
anytime.
We
learn
something
new
or
anything
like
that,
and
it
talks
a
little
bit
about
some
of
the
approaches
that
we're
going
to
talk
about
today.
A
So
one
of
the
main
considerations
is
people
are
planning
to
manage.
A
fleet
of
clusters
is
if
you
should
go
with
the
Hub
and
spoke
model.
That's
one
Argo
CD
instance:
managing
many
clusters
and
the
other
is
a
per
cluster
model,
should
I
have
an
Argo
CD
instance
in
every
cluster.
This
is
especially
popular
with
Edge
deployments,
where
a
cluster
needs
to
be
able
to
operate
independently,
with
potentially
bad
networking
conditions
where
things
might
get
pulled,
offline
or
whatnot,
and
a
per
cluster
approach
is
also
favored
if
you're
doing
an
air
gapped
instance.
A
So
if
you
have
like
somebody
that
needs
to
walk
a
repo
update
with
a
thumb
drive
to
some
secure
site,
or
something
like
that,
so
both
of
these
approaches
have
their
pros
and
cons.
They
both
have
their
use
cases
and
sometimes
I,
hear
people
debate
them
in
terms
of
which
one
is
better,
ultimately,
which
doesn't
really
make
sense,
because
it's
really
based
on
the
situation.
But
at
what
point
would
you
need
to
split
up
a
hub
and
spoke
because
you
might
need
multiple
right?
A
It's
not
just
a
matter
of
having
one,
because
in
adobe's
use
case,
you've
got
thousands
of
developers,
you're
onboarding
more,
and
you
have
these
additional
partner
development
teams.
You
have
to
support
so
there's
not
going
to
be
a
situation
where
you're
like
we're
just
going
to
run
one
Argo
instance
for
the
entire
company
right.
A
So
what
we
want
to
do
is
look
at
these
use
cases,
but
then
also
compare
it
against
Real
benchmarking
data.
So
to
do
this,
there's
actually
a
tool
hidden
in
plain
sight.
In
the
Argo
CD
repository
there
is
a
folder
called
hack
and
under
hack.
There
are
a
number
of
tools
that
we
use
to
do
benchmarking
and
performance
testing
of
Argo
CD.
So
when
we
make
changes
to
Argo
cd
part
of
the
automation,
we
actually
spin
up
resources
and
check
to
reconciliation
times
and
things.
A
So
we
can
just
make
sure
that
performance
isn't
impacted
by
a
particular
change,
or
we
can
understand
that
change.
So
under
the
hack
folder,
there
is
a
tool
called
generate
resources,
and
you
can
see
an
example
of
it
running
here
and
you
can.
You
can
specify
generate
X
number
of
applications
generate
X
number
of
clusters,
X
number
of
repositories,
X
number
of
projects,
and
these
each
of
these
points
hits
on
a
different
component
of
Argo
CD's
scale.
A
So
what
it
does
is
in
this
case
this
would
be
set
up
to
spin
up
200
V
clusters
and
then
assign
7
500
applications
randomly
across
those
clusters.
There's
another
tool
within
HACC
called
simulator
simulator
is
essentially
a
chaos
engine.
It
simulates
a
really
bad
set
of
developers
who
are
going
around
and
deleting
stuff,
and
so
you
can,
you
can
spin
up
simulator
and
it
will
go
around
and
mess
with
applications
and
Trigger
reconciliation
and
synchronization.
A
A
The
main
thing
isn't:
applications
number
it's
actually
the
number
of
objects
that
are
being
managed,
kubernetes
objects
because
for
each
object
we
have
to
reconcile
that
from
etcd,
and
that
takes
memory
and
it
also
hits
the
kubernetes
API,
and
so
this
takes
additional
overhead.
So
this
was
the
testing
plan
and
here's
a
screenshot
of
Prometheus
reporting
over
10
000
applications
and
going
through
this
exercise,
we
were
able
to
look
at
all
the
different
breaking
points
that
we
might
encounter
now
I'm
going
to
go,
I'm
not
going
to
go
through
all
of
them.
A
Now,
when
you
install
Argo
CD,
there
are
two
flavors
there's
Argo
CD
and
there's
Argo
cdha,
there's
actually
more
flavors
than
that,
but
Argo
cdha
replicates
redis
and
provides
the
replication
capability
and
scale
capability
that
we're
going
to
be
wanting
to
look
at.
So
some
of
the
things,
for
example,
that
I
ran
into
once.
I
got
up
around
7000
applications.
A
I
started
to
have
issues
just
with
my
browser
loading
the
dashboards
properly,
just
because
there
are
so
many
objects
in
memory
that
my
browser
was
having
a
hard
time,
keeping
track
of
it
as
I
moved
up
from
a
thousand
to
2500
applications.
I
noticed
that
my
mean
reconciliation
time
started
to
creep
up
and
my
reconciliation
cues
started
to
go
up
now.
This
is
very
solvable.
There's
actually
settings
in
Argo
CD
that
you
can
use
to
actually
solve
all
of
these
problems.
A
So
my
reconciliation,
cues
increasing
means
that
it
might
take
a
long
time
to
get
back
to
an
application
to
synchron
it.
So,
as
I've
increased,
you
might
get
into
a
situation
where
you're
waiting,
you
know
10
or
20
minutes
for
an
application
to
synchronize.
If
you
haven't
tweaked
or
managed
these
settings
properly.
So
this
this
gave
us
a
good
Baseline
of
some
of
the
breaking
points
that
we
might
hit.
A
A
So
we'll
see
so
this
is
very
conservative,
because,
if
you're
planning
to
support
you
know
40
000
developers
or
something
you
just
want
to
have
a
benchmark
in
mind
of
what
should
I
expect
where
I
just
don't
have
to
touch
anything
and
I'm
just
going
to
know
it's
going
to
work
and
at
what
point
do
I
need
to
start
thinking
about
that
scale.
Aspect.
B
A
B
A
The
the
1500
app
line
here
is
really
based
on
the
fact
that
most
people
have
more
than
seven
objects
in
an
app.
They
mostly
have
you
know:
100
objects
or
200
objects,
so
the
object
count
is
really
the
driver.
There,
15
000
objects,
you're,
going
to
be
very
safe,
you're
going
to
feel
very
comfortable
you're,
not
going
to
be
really
running
into
any
issues.
50
clusters-
you're,
not
gonna,
have
any
issues.
If
200
developers
are
running
around
and
doing
stuff
all
the
time
you're
not
going
to
have
any
issues.
A
So
this
is
a
very
safe
Benchmark.
Now
with
tweaking,
we
can
go
far
beyond
this.
Now,
when
I
talk
about
going
Beyond,
this
I'm
not
talking
about
catastrophic
failure,
I
mean
when
when
I
was
I
was
running,
10
000
applications,
they
were
synchronizing.
Sometimes
it
would
be
slow.
Sometimes
it
might
crash
and
restart,
but
it
would
pick
up
and
run
again.
So
it
was
fine.
So
it's
doable
it's
just
a
matter
of
tweaking
it,
so
you
actually
have
a
really
good,
reliable
experience.
B
So
like
this
is
the
part
where
I
get
kind
of
curious
about,
because
it's
an
area
where
I'm
kind
of
challenged
at
I
have
workloads
that
you
know
require
isolation.
I've
been
thinking
about
like
do
you
know
how
much
do
I
have
to
kind
of
like
how
many
Argo
CD
instances
do
I
going
to
need
to
kind
of
support
these
environments?
What
would
you
find
like
with
like
security
side
of
it.
A
Yeah
one
of
the
interesting
use
cases
that
you
have
is
that
you
need
to
support
not
only
multiple
vpcs,
which
means
I'm,
probably
going
to
have
multiple
Argo
instances,
because
I
don't
want
to
punch
through
firewalls
in
most
of
those
cases,
they're
vpcs.
For
a
reason.
You
also
have
these
on-prem
instances.
You
also
have
Edge
clusters,
so
those
those
are
going
to
take
into
consideration
potentially
additional
instances,
and
then
you
also
are
supporting
Partners
now
the
partner
one
is
an
interesting
one,
and
it's
one
that
most
people
aren't
aware
of.
A
So
you're
saying
add
arbitrary
code
execute
arbitrary
code
on
our
space.
Now,
we've
done
a
lot
of
work
to
make
sure
that
it's
very
difficult
to
do
things
like
grab
secrets
from
the
cluster
or
output,
those
things
and
there's
a
lot
of
guardrails
to
protect
that,
and
so
it's
considered
very
safe
within
an
organization
for
you
to
use.
A
However,
we
would
not
put
an
Argo
CD
instance
out
on
the
internet
and
just
say:
here's
multi-tenancy
everybody
can
get
to
go
deploy
to
our
namespace,
and
this
is
actually
something
that
came
up
for
us
at
code
fresh
because
part
of
our
offering
includes
a
hosted
Argo
CD
instance
that
is
like
on
every
account
every
you
every
account
gets
one
and
so
for
us.
We
actually
also
took
the
v-cluster
route.
A
We
spin
up
a
v
cluster
for
every
new
account
and
we
put
Argo
CD
on
it,
because
we,
while
the
rbac,
is
excellent,
we
don't
consider
the
rbac
and
as
a
project
we
don't
consider
our
Argo
CD
safe
to
use
in
a
public
multi-tenant
environment
like
that.
So
where
you
have
Partners
that's
going
to
be
an
impact.
A
Conway's
law
generally
prevents
a
lot
of
people
from
running
into
these
issues
early,
because
one
team
is
spinning
up
on
Argo
seating,
since
another
team
is
spinning
up
at
inargo
CD
instance,
and
as
many
organizations
mature
they're
going
to
start
thinking
about
okay,
how
do
I
make
sense
of
all
of
the
instances
that
are
running?
This
is
a
good
time
to
start
thinking
about
something
like
a
control
plane
in
the
interest
of
full
disclosure.
A
This
is
something
that
code
fresh
sells,
so
I'm
not
going
to
make
a
big
sales
pitch
to
you,
but
it
allows
you
to
manage
and
track
versions
of
all
of
the
instances
you
can
have
a
single
UI
for
all
of
your
applications
and
all
the
Clusters.
You
get
better
security
isolation
because
the
cost
of
it
splitting
up
instances
is
lower,
and
then
you
just
have
better
resiliency
and
lower
lower
blast
radius.
So
it's
something
to
think
about.
A
As
you
expand
up
some
other
considerations
are
most
people
move
to
a
two
repo
model
once
they
use
Argo
CD,
that's
something
we
recommend
where
you
have
application
repos
over
here
and
then
you
have
your
get
Ops
repo
or
get
Ops
repos.
That
is
defining
what
should
be
deployed
and
that
oftentimes
means
that
multiple
teams
are
operating
in
the
same
git
repo,
which
is
not
something
that
organizations
are
used
to,
and
so
that's
when
something
like
a
code
owner's
file
becomes
very
valuable
code
owners
files.
A
Allow
you
to
set
permissions
within
a
git
repository
based
on
folders,
so
you
can
say
this
team
can
deploy
to.
You
can
add
stuff
to
that
folder.
This
team
can
add
stuff
to
that
folder
and
that's
something
that
most
organizations
aren't
familiar
with.
Open
source
organizations
are
very
familiar
with
it
because
we've
been
using
it
for
a
long
time,
but
something
to
consider
and
then
also
getting
get
Ops
certified.
There
is
a
training
for
this
yeah.
A
Or
five,
so
this
is
currently
free
at
learning.codefresh.io,
so
it's
worth
grabbing,
it
won't
be
free
forever.
So
what
does
all
this
mean
for
Adobe?
If
every
pull
request
is
generating
the
cluster?
It
means
that
we're
going
to
need
to
replicate
the
Argo
CD
app
controller.
This
is
a
built-in
feature
of
Argo
CD
to
help
you
scale
it's
basically
a
stateful
set
that
is
going
to
divide
up
and
Shard
how
we
manage
all
of
the
Clusters
and
the
reconciliation
for
the
scale
that
we're
looking
at
over
probably
the
next
year.
A
This
is
going
to
be
very
sufficient
for
what
we're
going
to
be
doing
as
we
go
beyond
that
and
and
we're
supporting,
potentially
above
thousands,
potentially
getting
up
to
like
the
10
000
different
environments
range.
We
might
want
to
think
about
actually
splitting
up
instances
into
multiple
at
that
point.
A
If
every
generated
cluster
generates
a
PR
and
we
also
have
an
app
per
cluster,
we
need
to
keep
an
eye
on
object
counts.
If
you
try
to
run
a
thousand
objects
on
one
application,
you're
probably
going
to
have
some
reconciliation
challenges.
So,
to
solve
this,
it's
good
to
use
an
app
of
apps
pattern
where
you
basically
split
up
the
application
into
multiple
components,
and
this
has
other
advantages,
because
you
can
do
things
like
say
this.
A
B
A
B
B
So
this
kind
of
really
means
that
you
know
we
kind
of
a
plan
we
have.
We
have
this
Fleet
in
motion,
and
so
you
know
looking
at
these
numbers,
we
know
there's
some
opportunities
and
a
lot
of
strategies
that
I
think
a
lot
of
us
can
kind
of
Leverage
and
take
from
there,
but
as
far
as
like,
where
we're
going
with
the
Argo
CD
and
that
infrastructure
at
least
there's
kind
of
a
plan
to
know
how
to
scale
this
and
trying
to
do
it.
A
B
A
C
groups:
the
prequel,
the
cigarettes
is
the
pre.
Well,
there's
there's
a
whole
prequel
series
from
get
off
strong,
so
there's
even
a
prequel
from
there
so
check
out
the
extended
Universe
of
of
adobe
talks.
We've
got
a
whole
bunch
of
resources
that
I'll
put
these
slides
up.
These
will
be
uploaded
into
Coop
into
the
scheduler
by
the
end
of
the
day
and
I'll
tweet
a
link,
and
then
you
can
follow
me
at
today
was
awesome
and
if
you
have
any
questions
we'll
be
in
the
hall.