►
From YouTube: Unveil the Secret Ingredients for Argo CD in the Enterprise-Scale - Hong Wang & Yuan Tang, Akuity
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
揭开企业级 Argo CD 的构成秘密 | Unveil the Secret Ingredients for Argo CD in the Enterprise-Scale - Hong Wang & Yuan Tang, Akuity Inc.
与电影《功夫熊猫》中的面汤毫无秘密配方不同的是，大量优质研究已注入企业级 Argo CD 产品之中。您知道 Argo CD 可支持数以千计的应用程序吗？您是否曾经尝试过连接数以百计的 Kubernetes 集群？单个应用中存在上千个对象的情况如何？我们将深入研究 Argo CD 产品，向您提供解答和最佳实践。此外，我们还会分享和讨论一些与其他 Argo 产品有着良好协同的案例。
Unlike the fact that there is no secret ingredient in Kung Fu Panda’s noodle soup, a good amount of effort has been baked into the enterprise-scale Argo CD product. Do you know Argo CD can support thousands of applications? Have you tried to connect hundreds of Kubernetes clusters? What about the case with thousands of objects in a single application? We will deep dive into the Argo CD product, bring answers and best practices to you. In addition, several great synergy examples with other Argo products will be shared and discussed.
A
A
A
A
A
A
A
A
A
A
With
a
lot
of
scalability
issues
resolved
in
the
recent
releases,
we
will
shift
more
energy
to
integrate
ecosystem
projects
in
including
application
set
notifications,
image
updater
into
the
main
product.
Those
projects
have
been
tested
and
validated
for
a
while.
We
believe
they
are
mature
enough
to
be
admitted
to
enhance
the
out-of-box
rlcd
experience.
A
B
Before
we
dive
deep
into
the
scalability
challenges,
let's
talk
about
githubs
in
general.
First,
what
is
githubs
one
of
the
definitions
is
githubs
is
a
set
of
practices
to
manage
infrastructure
and
application
configurations
using
git.
That
means
any
detox
operator
needs
to
automate.
The
following
steps:
in
sequence:
first
retrieve
manifests
from
git
by
cloning.
The
git
repository,
such
as
from
guitar
or
bitcoin.
B
Second,
compare
git
manifests
with
live
resources
in
the
kubernetes
cluster,
using
control.
Div,
finally,
use
quick
control
apply
to
push
changes
into
the
kubernetes
cluster.
This
is
exactly
what
argo
cd
is
doing.
The
github's
workflow
does
not
seem
to
be
too
difficult.
However,
the
devil
is
in
the
details:
let's
go
ahead
and
find
out
what
can
go
wrong
and
what
can
you
do
about
it?
B
B
B
The
reason
is
that
argo
cd
is
delivering
github's
functionality
as
a
service
to
multiple
teams.
It
is
able
to
manage
multiple
clusters,
which
retrieve
manifests
from
multiple
git
repositories
and
service.
Multiple
independent
teams-
in
other
words,
you
can
enable
githubs
for
application
engineers
in
your
company
without
having
to
ask
them
to
run
and
manage
any
additional
software.
B
B
B
B
B
First
ago,
cd's
controller
runs
with
multiple
workers
form
a
pipeline
that
reconciles
applications,
one
by
one
in
sequence.
The
default
number
of
processes
is
20.
This
is
typically
enough
to
handle
hundreds
of
applications.
However,
if
you
get
a
thousand
or
more
applications,
you
might
start
seeing
a
few
hundreds
of
milliseconds
delay.
B
One
strategy
to
improve
the
performance
and
reduce
delay
is
to
increase
the
number
of
workers
in
the
controller
you
can
modify
it
in
controller
status.
Processors
inside
your
argo,
cd,
configmap,
larger
number
of
workers
means
that
argo
cd
will
be
processing
more
applications
at
the
same
time.
Note
that
this
would
also
require
more
memory
and
cpu
resources,
so
don't
forget
to
update
your
controller
resource
requests
and
limits
accordingly,.
B
B
Unlike
stateless
web
applications,
it
is
impossible
to
just
run
multiple
instances
of
kubernetes
controller.
The
challenging
part
for
argo
cd
is
that
controller
needs
to
know
the
state
of
the
whole
managed
kubernetes
cluster
properly
to
properly
reconcile
application
resources.
However,
you
may
be
running
multiple
controller
instances
where
each
instance
is
responsible
for
a
subset
of
kubernetes
clusters.
You
are
managing,
so
sharding
can
be
enabled
by
increasing
the
number
of
replicas
in
august.
The
application
controller
stateful
set,
don't
forget
to
update
the
rcocd
controller
replicas
environment
variable
with
the
same
value.
B
B
The
next
component
that
might
require
tuning
is
argo.
Cd
repo
server,
as
I
mentioned
earlier,
the
report
server
is
responsible
for
retrieving
the
resource,
manifest
from
the
git
repository.
That
means
argo
cd
needs
to
clone
the
repository
and
retrieve
yaml
files
from
the
cloned
repository
cloning.
The
git
repository
is
not
the
most
challenging
task.
One
of
the
github's
best
practices
is
to
separate
application
source
code
and
deployment
manifest,
so
the
deployment
repositories
are
typically
small
and
don't
require
a
lot
of
disk
space.
B
The
problem,
however,
is
that
deployment
repository
usually
don't
have
plain
yama
files.
Instead,
users
prefer
to
use
config
management
tools
such
as
customized,
helm
and
jsonnet.
These
tools
help
developers
to
avoid
duplicating
yaml
content
and
allows
to
introduce
changes
more
effectively.
Of
course,
you
might
ask
other
users
to
store
generated
yaml
in
deployment
repository
but
ago.
Cd
has
a
better
solution.
It
can
run
manifest
generation
on
the
fly
ago.
B
Typically
running
three
to
four
report:
server
instances
is
enough
to
handle
hundreds
or
even
thousands
of
repositories
ago,
cd
aggressively
caches
generated
manifests.
I
don't
need
to
regenerate
manifest
frequently.
However,
you
might
encounter
some
performance
issues
if
you
store
deployment
manifests
in
so-called
mono
repositories.
A
mono
repository
is
a
repository
that
simply
has
a
lot
of
applications.
B
The
real
world
model
report
might
have
hundreds
of
applications,
including
infrastructure
components
as
well
as
multiple
microservices.
Typically
mono
repositories
are
used
to
represent
desired
state
of
an
entire
cluster.
This
causes
the
following
performance
challenges:
first,
each
commit
to
the
monoreport
invalidates
existing
cash
for
all
applications
in
that
repo.
B
That
means
aggro
cd
needs
to
suddenly
regenerate
manifest
for
hundreds
of
applications
which
causes
cpu
or
memory
specs.
Second,
some
config
management
tools
does
not
allow
concurrent
manifest
generation.
For
example,
multiple
applications
that
rely
on
a
hem
chart
with
conditional
dependencies
have
to
be
processed
sequentially.
B
B
Next,
I'd
like
to
introduce
another
performance,
optimizer
station
technique
that
might
help
you
avoid
manifest
generation.
Spikes
completely
argo
cd
invalidates
manifest
cache
for
all
applications,
since
it
does
not
assume
that
the
generated
manifest
depends
only
on
files
within
application
related
directory.
However,
this
is
often
the
case
in
order
to
avoid
unnecessary,
invalidating
cache.
B
When
unrelated
files
are
changed,
you
can
configure
commit
webhooks
and
annotate
auto
cd
applications
with
the
auto
cd
manifest
generate
path.
Annotation.
This
annotation
value
should
contain
a
list
of
directories.
The
application
depends
on
so
every
time
when
webhook
notifies
argo
cd
about
a
new
commit,
it
will
inspect
the
changed
files
listed
in
the
webhook,
payload
and
reuse
any
generated
manifest
from
a
previous
commit
if
new
commit
does
not
touch
any
related
files
to
that
application.
B
The
api
server
is
a
stateless
api
that
scales
well
horizontally
and
does
not
require
too
much
computational
resource.
The
api
server
keeps
on
in
memory
cache
for
all
auto
cd
applications.
So
if
you
are
managing
more
than
5000
applications
using
one
argo
cd
instance,
you
might
want
to
consider
adding
additional
memory
limit.
B
Argo
cd
also
exposes
numerous
prometheus
metrics.
For
example,
accuracy.
The
app
reconcile
metric
indicates
the
application
reconciliation
performance
work
queue.
Depth
metric
indicates
the
depth
of
the
controller
queue
and
then
argo
cd,
app
sync
total
counts.
The
number
of
application
sync
operations
in
history.