►
From YouTube: ArgoCon '21-Argo-based service delivery for multi-tenant, multi-region clusters at Adobe(Aya Ivtsan)
Description
How do you enable CI/CD for hundreds of services, each with its own pipeline requirements and use-cases, all deploying to multiple clusters in multiple regions? At the Adobe platform team, we are using Argo Workflows, Events, CD, and Rollouts to create a flexible, customizable developer experience for CI/CD, where developers get a secure end-to-end pipeline out of the box and are able to tweak and modify it in a self-serve manner. Join us to learn how we enable and empower teams to manage their own Argo-based CI/CD pipelines across multiple multi-tenant clusters.
A
B
Great
to
be
here
all
right,
hi
everyone
good
morning
or
good
evening,
depending
on
where
you
are
very
excited
to
speak
here
at
the
very
first
arbolcon
congrats
to
the
argo
developers
and
to
the
organizers
and
thank
you
for
bringing
us
all
together.
My
name
is
aya.
I've
been
working
at
adobe
for
the
past
15
years,
and
I've
been
here
through
adobe's
transition
from
the
desktop
to
the
cloud.
I'm
working
on
our
cloud
foundation,
platform
called
ethos.
B
These
product
services
are
using
several
platforms
like
content
platform,
data
platform
and
sensei
machine
learning,
platform
and
all
services,
both
product
services
and
the
platform
services
use
ethos,
which
is
our
cloud
foundation.
Platform
and
ethos
itself,
has
clusters
running
on
aws
azure
and
data
centers.
B
B
There
are
around
26
000
tenant
name
spaces
on
those
clusters
and
in
those
there
are
over
150
000
workloads
running
at
any
given
time.
We're
averaging
at
around
3
million
containers
per
24
hours
and
we're
doing
around
800
builds
a
day
and
around
200
deployments
throughout
through
our
cicd
solution
per
day.
B
B
B
The
platform
is
cloud
agnostic
and
its
main
purpose
of
the
platform,
in
addition
to
the
management
of
kubernetes
cluster
and
everything
that
entails
is
mostly
to
handle
cross-cutting
concerns.
So,
for
example,
we
have
blessed
based
images
that
developers
base
their
own
images
off
of.
We
have
out
of
the
box
code
generation
and
service
that
and
service
bootstrapping
for
java
services,
which
comprise
around
70
percent
of
all
adobe
services.
B
We
offer
a
framework
and
libraries
to
do
things
like
easily
connect
to
other
adobe
services,
logging,
validations
and
things
like
that.
We
also
offer
kubernetes
namespace
provisioning
with
bootstrapping
of
things
like
secure
network
policies,
quota
limitations,
we
have
a
cicd
solution
and
more
on
this.
Coming
up.
B
B
B
B
So
now
we
have
two
offerings
for
adobe
developers.
We
have
the
paved
path
and
the
do-it-yourself,
which
we
can
also
think
of
as
the
wild
west
option,
with
a
paved
path
developers
get
full
abstraction
of
kubernetes
full,
automated
provisioning,
opinionated
cicd
with
guardrails
out
of
the
box,
but
very
little
or
very
limited
flexibility.
B
We'll
talk
about
what
this
means
in
the
next
slides,
whereas
with
the
do-it-yourself
option,
developers
get
maximum
flexibility,
but
they
have
to
bring
their
own
cicd.
They
get
only
namespace
provisioning
and
there's
no
abstraction,
meaning
developers
have
to
have
a
very
good
knowledge
and
understanding
of
kubernetes.
B
A
bit
more
about
our
paved
path
or
1.0
cicd
solution,
which
has
been
with
us
since
the
mesos
days.
It
is
implementing
the
evergreen,
the
evergreen
main
branch
philosophy,
meaning
that
changes
get
merged
to
the
main
branch
only
after
they
are
validated
in
production,
as
opposed
to
githubs.
As
you
all
know,
where
changes
get
merged
to
main
branch
as
a
trigger
to
the
automated
deployment
using
the
evergreen
main
was
key
for
us
at
the
time
as
guardrails
to
encourage
adobe
developers
to
do
frequent
deployments,
while
avoiding
things
like
git
revert,
risks
and
messy
situation.
B
It
was
also
allowing
them
to
do
easy
and
safe
rollbacks
other
all
other
guardrails.
We
have
include
enforcing
code
reviews,
sequential
environment
promotion,
image
scanning
and
more,
and
we
have
the
abstraction
that
I
mentioned
over
the
underlying
technology.
It's
basically
a
yaml
spec,
where
you
describe
your
service
configuration
needs
so
that
that's
our
current
1.0c
icd
solution.
B
B
B
Some
examples
for
the
unsupported
features
include
cascade
deployments,
multiple
pipelines
per
service
and
more
and
because
we
are
using
the
full
abstraction
without
a
way
for
tenants
to
override
it.
Any
new
feature
needs
to
be
made
available
through
that
abstraction
in
our
multiple
ci
cd
tools,
even
if
it's
already
available
out
of
the
box
on
the
kubernetes
cluster,
this
is
causing
slow
progress
for
adding
new
features
and
is
forcing
more
and
more
developers
to
turn
to
the
diy
offering
and
the
diy
offering
having
more
teams.
B
B
B
B
B
B
B
We
are
using
this
hub
cluster
for
running
argo,
workflows
and
events
in
tenant,
specific
namespaces
and
the
hub
cluster
also
has
argo
cd
running
in
it,
which
is
remote
connected
to
all
the
clusters
in
our
fleet
on
each
of
the
remote
clusters.
We
have
argo
rollouts,
so
we
are
implementing
this
kind
of
hub
and
spoke
model
for
the
clusters.
B
B
B
B
So
a
developer
makes
the
code
change
to
the
app
code.
This
in
turns
triggers
an
argo
workflow.
Through
our
events,
the
workflow
has
steps
to
first
build
the
image,
scan
it
and
push
it
to
an
image
repository
next,
there's
a
promotion
to
the
environment,
and
that
is
done
by
updating
the
image
version
in
the
desired
namespace
location
in
github,
which
in
turns
triggers
the
argo
cd
sync.
B
These
steps
from
the
promote
promote
to
n1
until
notify
can
be
repeated
for
each
environment
and
region
sequentially
or
in
parallel
on
the
remote
namespace.
If
our
rollouts
objects
are
defined,
which
they
are
by
default,
the
deployment
will
be
going
through
a
progressive
delivery
strategy
such
as
canary
or
canary
with
analysis,
which
can
be
defined
per
service.
B
B
We
implemented
a
wait
for
deployment
step.
We
implemented
an
active
weight
as
a
starting
point
since
at
the
time
when
we
started
with
this
argo,
cd
notification
was
not
yet
in
1.0
version,
but
we
do
plan
to
use
argo
cd
notification
with
workflow,
suspend
and
resume
pattern,
and
we're
also
investing
in
training,
we're
adding
self-paced
training
and
documentation
to
help
service
developers
with
adobe
specific
parts
of
the
solution,
while
directing
them
to
public
docs.
For
the
argo
staff.
B
So
a
bit
more
about
the
default
workflows
with
guardrails
those
are
generated
out
of
the
box
for
developers.
As
I
mentioned,
all
common
steps-
reference
cluster
workflow
templates
all
manifests
are
helmified
and
synced
to
the
hub
cluster
with
argo.
Cd
default
workflows
are
configured
through
values,
files
with
sequential
environment
promotion
and
parallel
deployment
to
regions
by
default,
and
the
default
workflows
include
things
like
image,
scanning
validation,
step
placeholders
for
integration,
step
tests.
B
B
The
helm,
charts
that
define
a
service
or
application?
Namespaces
are
managed
centrally
in
helm,
repository
by
us
they're
referenced
from
developer
repos
as
home
dependencies
and
specific
features
such
as
scaling
parameters,
ingress,
environment
variables,
etc,
are
all
controlled
through
the
values
files,
the
values
files
have
hierarchy
for
environments
and
regions
and
as
part
of
the
default
work.
B
B
B
On
the
left,
we
have
a
git
repo
for
the
shared
template,
which
is
maintained
by
us,
the
platform
team
that
repo
has
a
helm
chart
for
the
shared
templates
and
two
desired
state,
one
for
the
stage
cluster
and
one
for
the
prod
cluster.
Those
are
synced
to
the
station
prod
hub
clusters
respectively,
with
argo
cd.
B
B
So
our
journey
to
argo
based
cicd
solution
is
currently
on
the
way
some
of
the
things
we're
planning
to
tackle.
Next
we're
productizing
the
hub
clusters
and
rbos
services,
we're
enabling
more
self-service
like
adding
automation
for
failure,
recovery,
automation
for
provisioning,
both
for
the
initial
provisioning
and
ongoing
changes
such
as
adding
an
environment
or
a
region
we're
thinking
of
a
unified
ui,
where
developers
will
be
able
to
see
what
are
all
the
pipelines
and
namespaces
for
their
service.
B
B
Some
takeaways
have
a
modular
ci
cd,
give
developers
secure
paved
path
out
of
the
box,
but
make
sure
to
enable
a
way
to
customize
and
override
the
paved
path
in
order
to
enable
fast
progress,
make
it
easy
and
safe
to
contribute
to
the
platform
as
open
development
leverage.
The
industry
build
just
enough
on
top
of
it
and
contribute
contribute
back,
invest
in
kubernetes
native
tools
and
leverage.
The
rich
ecosystem
of
kubernetes
abstraction
is
important,
but
it
must
be
possible
to
override
it.
B
So
argo
is
a
great
fit
for
our
requirements,
we're
building
the
missing
the
pieces.
We
need,
on
top
of
it,
to
make
our
solution
work
and
we're
we're,
including
the
including
the
planned
customizable
abstraction,
which
we
are
planning
to
work
on
we're
looking
forward
to
working
with
all
of
you
in
the
argo
community
to
make
it
better
for
all
of
us.
B
You
know
if
you
have
a
custom
domain
name
that
you
need
a
custom
certificate
for
you
will
need
the
secrets
for
the
certificate,
the
secret
for
the
certificate
key,
etc.
So
all
those
runtime
secrets
require
secrets
on
their
remote
clusters
where
the
service
is
running
and
we
have
vault
operator
for
that.
B
So
vault
operator
is
as
a
service
that
is
running
in
the
tenant
name
space
and
it
is
controlled
through
crds
that
are
called
wall
secrets
and
whenever
vault
secrets
get
created
on
the
in
the
name
space,
the
vault
operator
translates
that
into
a
kubernetes
secret.
So
the
basically
the
helm
chart
defines
the
secrets
in
terms
of
vault
paths
and
field
names
and
then
those
get
translated
by
the
vault
operator
to
an
actual
kubernetes
secret
and
we're
doing
similar
thing
on
the
hub
cluster.
B
For
things
like
build
time
secrets-
or
you
know,
you
need
a
secret
to
push
the
image
to
your
image
repository.
We
are
one
other
thing
which
is
different
between
our
original
ci
cd
and
the
new
one
is
that,
with
the
new
ci
cd,
we
want
to
keep
all
the
secrets
to
be
provided
by
tenants.
In
other
words,
we
don't
want
to
share
any
secrets
across
tenants,
which
is
something
we
had
to
do
in
our
previous
solution
because
of
some
technical
limitations.
B
B
That
means
we
also
need
secrets
on
the
hub
cluster
for
things
like
the
build
step
or
the
image
push
step
and
again,
we
are
using
vault
operator
for
that
and
what
we
plan
to
do,
or
what
we
are
actually
working
on
actively
in
this
sprint,
is
to
add
the
vault
operator
into
our
default
helm
chart.
So
it
will
be
very
easy
for
developers
to
incorporate
the
vault
operator
in
their
both
hub
namespace
and
the
remote
namespace.
A
Think
so,
I
think
so,
and
I
know
we're
right
at
times,
so
I
wanted
to
give
you
a
shout
out
since
you
haven't,
read
the
chat,
yet
it
was
from
thomas
sebastian
and
he
said
that
this
is
the
best
talk
in
a
long
time
out
of
a
lot
of
conferences.
So
the
community
is
loving.
This
feel
free
to
jump
in
there
and
have
a
chat
with
them
and
those
of
you
that
are
still
listening.
Let's
get
back
in
the
chat
and
keep
this
conversation
going
and
we'll
see
you
on
the
next
one.