►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Multi-Tenancy For Argo Workflows And Argo CD At Adobe - Srinivas Malladi, Adobe
Speakers: Srinivas Malladi
Argo Workflows and Argo CD are powerful tools, but unifying them under a multi-tenant experience is necessary to run at scale across multiple teams in any large organization. Argo Workflows and Argo CD use different approaches to RBAC and both have different security considerations and available security features. We at Ethos, the Adobe Cloud Platform, have designed an architecture to create a secure multi-tenant CI/CD experience for our developer teams. Join our talk to learn how we achieved multi-tenancy through the isolation of each component of our developer CI/CD workflows, such as building, scanning, pushing, workflow artifacts, workflow secrets, as well as the restriction of application deployment with Argo CD AppProjects and RBAC.
A
A
Okay,
great,
let's
get
started
hello
and
welcome
to
my
talk
on
multi-tenancy
for
Argo
workflows
in
Argo
CD
at
Adobe,
I'm,
Serena,
vasmati
I'm,
a
software
engineer
at
Adobe
who
works
on
the
internal
developer
platform
at
Adobe,
so
before
I
get
started.
I
do
want
to
give
credit
by
emphasizing
that
the
work
I'll
be
preparing
today
is
not
just
a
product
of
my
own
efforts,
but
a
combined
product
of
efforts
of
many
of
my
colleagues
at
Adobe,
and
so
with
that
let's
get
started.
A
A
System
state
is
first
defined
and
tracked
and
get
somewhere
by
some
tooling.
That
tooling
then
deploys
that
defined
state
to
a
live
State
on
a
running
system.
The
githapsulin
regularly
synchronizes
the
two
states
so
that
the
live
State
can
get
automatically
pulled
in
with
any
updates
and
it
pulls
in
any
changes
made
inside
git.
A
Argo
CDE
is
an
example
of
one
such
git.
Ops
tooling.
It
supports
tracking
of
kubernetes
or
Kates,
manifests
in
git
and
supports
their
deployment
and
synchronization
to,
for
example,
a
namespace
on
a
cluster
Argo
CD
is
implemented
as
a
kubernetes
controller.
It
uses
various
crds.
One
of
them
is
called
an
Argo
CD
application
and
that
stores
information
about
what
defines
state
to
track
and
where
to
track
it
as
well
as
where
to
deploy
it.
So
here's
an
example
of
what
that
looks
like
the
yaml
Manifest
for
an
Argo
CD
application.
A
As
you
can
see,
there
are
sections
in
the
Manifest
defining
the
source
and
intended
deployment
destination
of
the
defined
State
and
Argo.
Cd
itself
supports
automated
deployment,
self-healing
complex
rollout
strategies,
monitoring
and
much
more
through
the
Argo
City
UI,
which
we
will
discuss
more
later.
The
Argo
CD
also
visualizes.
The
health
and
deployment
status
of
the
developers
deployed
resources.
Wherever
it's
deployed
on
the
cluster
Argo
workflows,
Argo
workflows
is
a
workflow
engine
that
can
run
CI,
CD
pipelines,
among
other
things,
on
acades
cluster.
A
There
are
many
different
kinds
of
workflows,
but
generally
a
workflow
can
be
thought
of
as
a
set
of
tasks,
and
each
task
runs
in
its
own
pod
and
can
also
be
modeled
as
a
dependency
on
other
tasks.
The
resulting
workflow
can
actually
be
templated
and
defined
using
yaml
files
like
this
on
the
screen,
and
this
demonstrates
a
directed
acyclic
graph
type
template
it's
an
example:
I
I
pulled
online.
A
The
use
of
templating
opens
the
door
to
sharing
a
template
to
be
used
as
a
sub
template
in
other
workflows,
since
a
workflow
is
implemented
in
a
kubernetes
crd
and
each
task
runs
in
a
pod.
For
example,
this
means
that
workflows
natively,
integrate
with
existing
kubernetes
objects,
such
as
volumes,
config,
Maps,
config
map
secrets
and
much
more
now,
let's
get
a
better
understanding
of
the
cicd
offering
that
IDP
at
Adobe
provides
and
what
it
looks
like
with
Argo
workflows
and
Argo
CD
working
together.
A
So
we
start
with
the
Hub
cluster,
where
we
have
Argo
workflows
and
Argo
CD
installed.
We
also
have
client
namespaces
where
client
workflows
are
run.
Additionally,
there
are
remote
clusters
with
remote
client
name
spaces
that
are
running
client
applications
alongside
any
case
resources
that
they
need
to
support
it.
These
remote
clusters
are
usually
pulling
an
image
from
some
registry
to
run
as
part
of
the
deployment.
A
Now
the
defined
state
for
all
of
this
is
actually
stored
in
git
and
using
git
Ops.
This
includes,
of
course,
client
repositories,
which
includes
the
application
code,
as
well
as
a
deploy
repository,
deploy.
Repository
contains
CI
CD
resources,
as
well
as
resources
associated
with
any
of
the
Kate's
resources
deployed
on
the
remote
namespace.
A
So
you
have
your
service
case
manifest
and,
of
course,
you
have
your
workflow
manifest,
which
are
the
cicd
resources
themselves,
so
these
repositories
may
also
refer
to
other
IDP
owned
resources
that
they
can
refer
to,
and
we
do
this
through
Helm
we'll
talk
more
about
that
later
as
well.
So
how
do
these
pieces
connect
so
to
start
with,
Argo
CD
is
always
tracking.
The
client
deploy
repository
manifests,
and
it's
doing
this
because
it's
constantly
deploying
them
to
the
workflows
on
the
Hub
cluster,
as
well
as
the
actual
deployed
resources
on
the
remote
cluster
itself.
A
Following
this
setup,
any
change
made
to
the
application
code
will
trigger
a
workflow
inside
the
client,
Hub
namespace.
And
specifically,
what
actually
happens
is
this
requests,
a
workflow
from
the
Argo
workflows,
controller,
that's
running,
and
the
Argo
workflows
controller
will
execute
a
workflow
in
namespace
that
requested
it.
A
So
one
of
the
steps
in
the
workflow
is
a
build
step
and
that
actually
builds
and
uploads
an
image
to
the
image
registry
that
we
mentioned
earlier.
The
other
step
after
that
goes
ahead
and
takes
that
tag
or
whatever
information
about
that
build
and
enters
it
into
the
service
case
manifests
so
that
it's
recorded
somewhere
in
git.
Now,
as
mentioned
earlier,
that
state
is
constantly
being
tracked
by
Argo
CD
and
so
that
information
is
then
deployed
to
the
actual
running
State
on
the
remote
cluster.
A
A
So
now
that
we've
gone
over
the
general
walkthrough,
the
general
architecture,
let's
take
a
look
at
how
we
make
this
multi-tenant
so
multi-tenancy.
It's
an
architecture
where
multiple
clients
are
sharing
a
single
resource.
In
this
case
it's
a
running
instance
of
Argo,
workflows
and
Argo
CD.
It
can
reduce
waste
by
reducing
idle
time,
as
well
as
a
number
of
running
instances,
depending
on
how
you
manage
load
and
how
scaling
is
handled.
It
can
save
developer
time,
mainly
because
it
allows
them
to
spin
up
faster.
A
A
It's
efficient
because
every
Improvement
or
feature
that's
made
to
the
shared
resource
is
a
force
multiplier
that
can
be
used
across
every
single
team.
That's
leveraging
it
and
because
of
all
of
this,
it
scales
really
well
because,
by
reusing
previous
work
relating
to
setup
and
maintenance,
we
are
reducing
the
amount
of
friction
it
takes
for
a
client
to
get
up
and
get
started.
A
Some
of
the
questions
we
had
to
answer
when
addressing
multi-tenancy
with
Argo
workflows
in
argocu
were
how
do
we
make
sure
that
clients
can
only
access
what's
theirs?
How
do
we
isolate
client,
workflows
and
processes
and
deployments?
And
what's
the
best
way,
to
get
updates
and
fixes
out
to
clients
for
resources
that
we
don't
control?
A
And,
of
course,
how
do
we
design
for
reliability
as
load
increases?
So,
let's
start
with
access
for
Argo
workflows.
This
page
shows
the
Argo
workflows
UI,
it's
exposed
by
the
Argo
server,
that's
part
of
the
workflows
installation
on
the
Hub
cluster
and
it
visualizes
client
workflows
that
run
inside
client
namespaces.
Based
on
our
architecture.
A
We
can
see
here
that
workflows
are
being
returned
for
a
specific
namespace
that
we
entered
and
also
that
a
workflow
is
currently
in
progress
after
clicking
into
a
workflow.
A
client
can
view
workflow
tasks
and
logs,
as
well
as
execute
actions
like
suspending
retrying
or
even
deleting
workflows,
and
while
this
is
great
functionality
to
have
how
do
we
make
sure
that
clients
are
only
doing
this
for
workflows
that
belong
in
their
own
namespace
on
the
cluster?
A
A
So
to
achieve
this,
we
use
single
sign-on
and
the
workflow's
namespace
rbac
feature.
Let's
go
through
this
step
by
step,
so
after
receiving
a
client's
request
to
access
the
workflow
resource,
the
Argo
server
will
first
get
the
group
member
information
for
that
client
through
the
configured
identity
provider.
In
our
case,
that's
Azure,
active
directory.
The
Argo
server
then
goes
ahead
and
checks
whether
a
service
account
is
associated
with
any
of
those
groups
inside
the
namespace
of
the
requested
resource.
A
If
it
does,
and
the
service
account
has
the
necessary
permissions
to
access
that
resource,
then
the
access
is
granted
and
if
it
does
not,
the
access
is
blocked.
So
this
access
flow
allows
us
to
scope,
client
access
down
to
workflow
resources
at
the
namespace
level,
and
it's
using
service
accounts.
So
it's
highly
scalable,
since
it
leverages,
you
know,
case
related
rbac,
through
service
accounts.
This
allows
a
many-to-many
relationship
between
our
clients
and
namespaces
on
the
Hub
cluster.
A
The
namespace
rbac
feature
that
comes
with
Argo
workflows
requires
that
the
service
account
is
annotated
with
the
authorized
groups
that
are
allowed
to
leverage
it
for
access
to
the
resources
in
the
hub
cluster.
You
can
see
this
in
the
example
manifest
here
which
specifies
that
only
clients
in
group
a
can
use
a
service
account
for
Access
it's
in
a
list
by
default,
so
you
can
add
to
those
groups,
there's
also
a
precedence
field
and
that's
mainly
used
to
break
ties
for
service
accounts
that
might
Match
multiple
service
accounts
that
might
match
to
a
client.
A
The
Argo,
workflows,
command
line,
interface
or
CLI
is
also
exposed
by
the
Argo
server.
But
the
mechanism
for
scoping
access
here
is
slightly
different.
This
time,
instead
of
an
identity
provider,
the
Argo
server,
actually
delegates
authentication
in
our
architecture
and
in
the
general
Argo
workflows
architecture
for
SSO
to
the
sorry
for
the
CLI
to
the
cube,
API
server.
The
client
includes
a
bearer
token
in
their
request,
which
may
be
automatically
included
as
part
of
an
exported
cubeconfig
file
locally
or
as
part
of
a
token,
in
the
actual
CLI
request.
That's
going
out.
A
The
token
is
validated
by
the
case
API
server
and
if
the
token
is
valid
and
contains
the
necessary
permissions
for
the
requested
resource,
then
the
request
is
granted.
So
here's
an
example
of
what
a
successful
CLI
looks
like
when
I'm
querying
for
my
own
workflows
from
the
CLI.
As
you
can
see,
this
is
my
own
namespace
and
I'm
getting
back
my
workflows
and
here's
an
example
of
what
happens
when
I
try
to
query
for
workflows
in
the
CLI
that
I
do
not
have
access
to
using
that
flow.
A
Now
that
we've
covered
how
access
is
scoped
or
how
we
scope
access
in
the
workflows
UI,
let's
take
a
look
at
the
Argo
CD
UI
and
how
it
does
it.
The
Argo
City
UI
visual
visualizes
applications
that
track
and
deploy
State
defined
in
git.
The
goal
here
is
that
clients
should
only
be
able
to
view
their
own
Argo
CD
applications
through
the
UI.
A
So
in
the
Argo
CDU
you,
the
UI
and
CLI
use
the
same
mechanism,
this
time
to
scope,
client
access
for
resources
like
Argo
CD
applications
similar
to
workflows.
These
are
exposed
by
the
Argo
CD
server.
In
this
case,
Argo
City
also
has
other
resources,
in
addition
to
Argo
CD
applications
such
as
logs
connected
repositories
connected
clusters,
certificates,
as
well
as
app
projects.
A
The
app
project
resource
is
important
as
we'll
see
later,
because
it
can
be
used
to
scope
or
group
Argo
CD
resources,
which
can
then
be
used
to
scope
access
some,
but
not
all
Argo
CD
resources
can
be
grouped
together
into
an
app
project.
Access
to
those
resources
can
then
be
scoped
to
clients
who
have
access
to
the
on
the
the
parent
app
project
itself
for
resources
that
cannot
be
scoped
to
app
projects.
Access
is
usually
defined
using
a
global
config
map.
A
We
generally
don't
use
Global
rbac
in
our
architecture
for
client
access,
because
it
can
act
as
a
single
point
of
failure
it's
prone
to
typos.
On
the
other
hand,
the
app
projects
are
really
great
because
they
cannot
buy
design,
be
used
to
provide
access
for
resources
that
don't
fall
under
those
app
projects,
making
it
ideal
from
an
access
scoping
perspective.
A
After
setting
up
project
scoping,
a
client
can
request
access
to
a
resource
and
the
Argo
CD
server
gets
the
client's
group
information
from
again
the
configured
identity
provider,
which
again
here
for
Argo
CD,
is
azure
active
directory.
The
Argo
CD
server
then
checks
if
any
of
the
clients
groups
are
authorized
by
the
parent
project
of
the
requested
resource.
If
so,
then
the
request
is
granted,
if
not,
the
request
is
blocked.
A
So
now
that
we've
finished
covering
how
access
is
configured
for
our
workflows
in
Argo
CD,
let's
go
over
how
isolation
is
Achieve
for
the
respective
tools,
beginning
with
Argo
workflows.
So,
as
mentioned,
each
other
workflow
runs
in
a
pod
and
they
run
the
client
Hub
namespaces.
So
far,
we've
ensured
that
clients
can
only
access
their
own
resources,
but
how
do
we
make
sure
that
they
can't
access
other
resources
through
their
workflows
and
how
do
we
protect
their
workflows
from
other
clients?
A
First,
we
require
that
all
clients
bring
their
own
secrets
to
the
namespace,
so
this
helps
with
workflow
access
automatically
scoping
it
through
automatically
scope,
git
tokens,
our
Factory
credentials,
Argo
CD,
automation,
tokens
and
any
other
secrets
that
we
pull
from
vault.
Next.
We
leverage
the
lack
of
shared
secrets
to
configure
individual
client
artifact
repositories,
and
this
adds
another
layer
of
isolation
for
the
artifacts
that
are
passed
between
steps
in
a
workflow.
A
With
regards
to
the
executor,
that's
actually
running
the
containers
inside
the
workflow
pods,
we
allow
the
recommended
Emissary
executor
by
default,
as
well
as
the
Kate's
API
executor.
The
latter
requires
some
Network
policies,
but
we
lock
that
down
by
using
default
Network
policies
to
control
for
Ingress
and
egress
by
default
when
triggering
workflows,
we
specify
a
service
account.
That
is
not
the
default
service
account
to
run
under
this
defines
the
permissions
the
entire
workflow
runs
under,
or
rather
each
task
and
also
helps
us
be
intentional
about
what
permissions
we're
giving
to
the
workflow
itself.
A
We
also
have
cluster
level
pod
security
policies
that
limit
pod
capabilities
and
force
default
setcom
profiles
and
prevent
privileged
pods
from
even
being
spun
up
on
the
cluster
itself
in
client
namespaces.
To
accommodate
this,
we
use
kaniko
to
build,
and
that's
because
Canada
code
does
not
require
any
special
privileges
in
order
to
build
our
our
client
images,
and
we
also
use
multi-stage,
builds
in
addition
to
that
so
for
clients
that
do
require
privileges,
for
example,
Docker
and
Docker.
A
So
the
result
of
these
layers
is
that
client,
workflows
and
the
underlying
pods
are
isolated
to
the
client
Hub
namespace,
in
which
they're
running
and
this
prevents
their
resource
from
being
accessed
by
other
clients
and
also
prevents
them
from
accessing
other
clients.
Resources
with
Argo
CD
isolation
has
to
happen
at
the
Argo
CD
application,
Level
first,
so
as
mentioned
previously,
an
Argosy
application
is
a
crd
that
can
be
deployed
to
specific
that
can
be
used
to
deploy
to
specific
namespaces
on
specific
remote
clusters.
A
The
rocd
application
is
also
configured
to
track
Kate's,
manifest
from
a
git
repository,
in
this
case
the
deploy
repository.
That's
owned
by
the
client
in
this
situation,
the
deploy
repository
might
be
referring
to
other
repositories,
for
example
in
Helm
charts
or
Helm
dependencies
in
artifactory.
So
how
do
we
restrict
what
a
client,
through
their
Argo
CD
application,
can
deploy
to
and
track?
The
answer
again
here
is
actually
Argo
CD
app
projects.
A
In
fact,
the
Argo
CDF
project
has
to
specifically
allow
tracking
of
git
locations
of
the
child
Argo
CD
applications
in
order
for
those
Argo
CD
applications
to
be
allowed
to
track
those.
Similarly,
it
has
to
specifically
allow
deployment
to
name
spaces
on
clusters
in
order
for
the
child,
Argo
City
applications
to
be
able
to
deploy
to
them.
A
If
the
Argo
City
application
does
not
follow
these
rules,
then
an
error
will
actually
be
thrown
when
attempting
to
create
the
application
or
put
it
inside
the
project
and
by
default
all
our
gocd
applications
have
to
be
inside
a
project.
Finally,
we
also
restrict
Argo
CD
applications
from
creating
any
cluster
level,
resources
like
creating
or
deleting
namespaces,
because
that's
not
good
when
you're
sharing
a
cluster
with
other
clients.
A
The
other
Argo
CD
component
that
we
look
at
with
regards
to
isolation,
is
the
argosity
operator
itself,
that's
running
on
the
Hub
cluster,
so
the
operator
is
connected
to
multiple
remote
clusters
that
it
deploys
through
a
hub
and
spoke
model,
and
this
means
that
the
operator
running
on
the
Hub
cluster
is
connected
and
deploys
to
all
of
these
possible
remote
namespaces
on
different
remote
clusters.
So
how
do
we
ensure
that
we're
appropriately
limiting
the
permissions
that
the
Argo
CD
operator
is
running
with?
A
So
I
was
CD
when
registering
a
cluster.
Actually
first
needs
to
have
a
service
account.
That's
on
the
cluster,
and
by
default,
this
service
account
is
created
in
the
registration
process.
In
the
cube
system,
namespace
with
admin
cluster
level
permissions
Argo
CD
is
then
given
the
token
associated
with
that
service
account
and
then
uses
that
token
to
manage
and
deploy
to
that
cluster.
Now
there
are
two
things
that
could
be
wrong
with
this
picture.
A
The
first
is
that,
by
default,
the
registration
process
out
of
the
box
puts
the
service
account
in
the
cube
system.
Namespace.
Now,
as
we
mentioned
earlier,
the
cube
system
namespace,
or
rather,
as
we
mentioned
earlier,
you
may
be
using
pod
security
policies
to
be
preventing
privileged
pods
from
being
spun
up
on
your
cluster.
A
lot
of
the
times
best
practice
is
actually
to
allow
list
service
accounts
inside
the
cube
system
namespace
so
that
they
actually
aren't
restricted
by
those
pod
security
policies.
A
So
in
order
to
begin
fixing
these
issues,
let's
start
with
the
cube
system,
namespace
issue,
so
the
cube
system,
namespace
issue.
If
you
just
change
the
namespace,
then
if
you
have
pod
security
policies
that
restrict
privileged
pods
being
spun
up
on
your
cluster,
then
it
will
work
as
expected
and
we've
restricted
Argo
series
ability
to
deploy
privileged
resources
and,
by
extension,
clients
that
use
Argo
CD.
A
It's
this
check
will
actually
error
out
if
it
does
not
have
a
cluster-wide
read
access
and
now
argocd
will
generally
be
fine
with
cluster
read
access
but
still
needs
admin.
Access
to
the
individual
namespaces
when
deploying
resources
to-
and
we
achieved
this
through
individual
role
bindings
in
each
of
the
namespaces-
give
the
service
account
right
access
to
the
namespaces
themselves,
and
this
limits
the
blast
radius
to
just
the
namespaces
that
are
supposed
to
be
managed
by
Argo
CD
themselves.
So
we
can
do
better.
A
Actually,
we
can
remove
Secrets,
especially
if
you
use
some
sort
of
Vault
integration
to
provision.
Kate
Secrets
separate
from
Argo
CD
on
namespaces,
so
argocity
does
not
need
to
read
secrets
to
the
cluster
level.
We
can
also
remove
secrets
from
the
right
access
that
it
needs
for
namespaces
itself,
because
again,
Argo
C
is
not
deploying
Secrets,
as
mentioned
earlier,
though,
how
do
we
prevent
the
cluster
bulk
rate
from
erroring
out
when
it's
trying
to
sync?
And
we
do
that
through
exclusions?
A
So
the
result
of
this
isolation
of
The
Argosy
operator
is
that
we
end
up
with
a
more
secure,
Hub
and
spoke
model
and
by
extension,
clients.
They
can't
do
as
much
on
the
connected
remote
clusters
that
are
shared
between
them,
so
we've
explored
access
and
isolation
for
Argo
workflows
in
Argo,
CD
and
I've.
Seen
that
clients
have
various
repositories
that
are
involved
in
line
with
the
git
Ops
model
that
we
follow,
clients
have
an
application
repository,
that's
running
application
code.
They
also
have
a
deploy.
A
Repository
that
contains
service
manifests
for
the
actual
running
service
and
Cates
resources
on
the
remote
cluster,
as
well
as
workflow,
manifest
for
the
cicd
pipelines
that
they're
running
on
the
Hub
cluster.
So,
unlike
the
shared
instances
of
Argo
workflows
and
Argo
CD,
these
resources
are
actually
client
owned.
So
how
does
IDP
help
distribute
fixes
updates
maintenance
to
these
client-owned
repositories
that
can
drift
over
time?
A
Well,
the
answer
here
is
that
we
try
to
centralize
what
we
can
and
so
for
the
application
code.
For
example,
we
provide
generation
templates
that
can
be
used
to
help
clients
spin
up
more
quickly
in
terms
of
their
application
code
for
the
CI
CD
resources
and
the
service
case
manifest.
We
use
our
own
repositories
and
we
connect
them
because
everything's
a
home
chart.
A
So
we
can
use
Helm
dependencies
to
automatically
pull
in
any
changes
that
we
deploy
to
our
centrally
managed
name
to
our
centrally
managed
repositories
themselves,
and
we
actually
use
semantic
versioning
in
order
to
achieve
this
in
a
more
seamless
way.
So
Helm
supports
semantic
versioning,
which
is
a
system
where
version
numbers
actually
convey,
meaning
about
things
like
backward
compatibility
and
breaking
changes
through
semantic
versioning,
client,
Helm,
charts
actually
configured
to
automatically
pull
in
minor
or
patch
releases
made
on
the
idp-owned
repositories.
Of
course,
minor
patch
releases.
A
A
Both
such
releases
made
for
IDP
Helm
charts
on
the
right
would
automatically
be
pulled
into
the
chart.
Helm
Helm
charts
on
the
left
due
to
semantic
version
integration.
This
is
powerful
because
it
allows
us
to
get
critical
security
updates,
fixes
best
practices
out
to
clients
immediately,
provided
that
these
are
all
Backward
Compatible.
A
A
This
requires
clients
to
manually
bump
the
version
of
the
IDP
Helm
dependency
that
they're
using
inside
their
own
repositories.
We
usually
try
to
include
release,
notes
on
the
breaking
change
and
once
clients
make
the
necessary
adjustments.
If
any
to
accommodate
these
changes,
they
can
immediately
pull
in
all
the
work
and
fixes
that
we've
made
inside
our
dependency
that
we
control
here's
an
example
of
a
major
release,
change
and
what
it
looks
like
moving
from
1.1.0
to
2.0.0
and
here's
an
example
of
what
a
client's
chart.yaml
looks
like.
A
That
shows
how
we
use
Helm
dependencies
with
semantic
versioning.
We
can
see
the
name
version
and
repository
that
the
IDP
owned
Helm
dependency
here
corresponds
to.
We
can
also
see
a
carrot
to
the
left
of
the
version
number
in
semantic
versioning.
This
means,
please
keep
me
updated
to
the
latest
compatible
version,
which
roughly
translates
to
what
we're
trying
to
do
is
please
pull
in
all
minor
or
patch
versions,
but
don't
automatically
pull
in
major
versions.
Let
me
do
that.
This
is
what
it
looks
like
in
a
client's
Repository.
A
So
as
we
build
out
multi-terrain
architecture,
how
do
we
prepare
for
heavy
load
that
may
appear,
as
we
scale
up
well,
for
high
availability?
The
Argo
workflow
documentation
is
actually
quite
great
and
it
details
improvements
such
as
pod
disruption,
budgets
and
minimum
replica
counts
for
the
Argo
server,
and
this
helps
generally
improve
the
components
inside
the
Argo
workflows
installation,
as
we
onboarded
more
clients
running
workflows
in
different
namespaces
on
the
Hub
cluster.
A
We
also
increase
the
workflow
executor's
default
resource
allocations
for
CPU
and
memory,
since
this
was
limiting
the
size
of
the
artifacts
that
clients
were
passing
through
or
between
workflow
tasks
themselves,
and
we
also
increased
the
default
main
container
resource
allocation
to
accommodate
more
resource
intensive
workflows.
The
main
container
is
the
the
containers
is
actually
running
the
task.
On
the
other
hand,
since
we've
removed
or
increased
these
limits,
we
also
needed
to
be
able
to
limit
what
a
client
workflow
itself
could
consume
on
the
cluster,
and
we
did
this
through
namespace
level.
A
Quotas
that
limit
the
impact
a
single
client
can
have
on
the
resources
being
shared
by
all
the
clients
on
the
Hub
cluster
running
workflows,
Argo
workflows
provides
Prometheus
metrics
relating
to
workflow
controller
State
and
also
provides
support
for
custom
metrics
at
the
workflow
or
template
level,
documentation
on
the
default
metrics
exposed
and
how
to
create
custom
ones
as
well
as
example.
Grafana
dashboards
are
available
online
as
well
and
they're
great.
A
A
As
we
onboarded
more
clients,
we
switched
to
using
GitHub
apps
to
register
repositories.
This
was
both
due
to
the
fact
that
GitHub
apps
have
higher
rate
limits
than
generic
users
or
personal
access
tokens
that
you
may
be
trying
to
clone
from,
for
example,
when
Argo
CD
is
tracking
stayed
in
git
and
also
because
it
actually
reduces
client
friction.
Instead
of
having
to
generate
a
deploy,
repo
clients
don't
need
to
do
anything
except
install
the
GitHub
app
on
their
repository,
and
we
already
have
the
GitHub
app
credentials.
A
So
we
can
go
ahead
and
use
that
to
clone
the
repository.
So
that's
one
last
step
for
clients
as
well
as
we
scale
up.
We
also
looked
into
issues
like
limiting
the
ability
for
an
individual
client
to
be
able
to
overwhelm
Argo
CD
resources,
for
example
with
a
very
large
repository
for
the
Argo
CD
repo
server,
and
we
also
looked
into
sharding
Argo
CD
operators
across
different
Hub
clusters
in
order
to
help
distribute
the
load,
as
we
start
scaling
with
more
clients.
A
A
That
scales
well
with
more
clients
and
keeps
clients
isolated
from
each
other,
provides
them
with
scoped
access
to
their
own
resources
and
helps
us
maintain
a
distributed
support
framework
that
helps
clients
continue
to
accelerate,
and
this
generally
supports
the
ability
of
the
internal
developer
platform
at
Adobe
to
create
a
CI
CD
experience
that
remains
flexible
for
different
use
cases
and
represents
a
force
multiplier
for
all
of
our
developer
teams.
Thank
you
for
your
time.
B
Hi
thanks
for
the
great
talk
just
a
quick
question
about
the
authentication
and
Azure
ad
I,
think
in
your
in
your
diagram,
I
caught
that
you
just
showed
the
the
user
presenting
to
workflows
a
bearer
token
when
they
log
in
I
was
curious.
How
does
that
process
the
authentication
process
work?
Where
did
they
get
that
token?
How
is
the
single
sign-on
done?
Yeah.
A
Yeah,
so
the
token
itself,
so
that's
what
the
CLI
acts.
Oh,
the
question
was:
where
do
users
get
the
token
in
order
to
access
the
Argo
workflows
CLI,
because
we
showed
this?
This
is
yeah,
because
this
is
the
UI,
but
we
showed
the
perspective
from
a
client's,
a
client's
perspective.
So
where
do
they
get
that
token?
So
two
ways
they
get
it
from
their
Cube
config,
which
the
internal
developer
platform
provides,
and
that's
one
way
so
they
have
this
Cube
config
that's
exported
locally
and
that's
present.
A
A
The
other
way
is
if
they
actually
have
any
Bearer
token,
if
they
have
a
service
account
token,
if
they
have
any
kids
token,
they
can
actually
include
it
in
the
CLI
request
as
an
additional
flag
right,
but
usually
the
first
use
case
is
the
more
common
one,
because
most
of
our
clients
have
q
config
yaml
files
available
that
are
exported
locally,
that
allow
them
to
access
the
cluster
itself.
There
are
scope
to
their
namespaces
yeah
great
question.
Thank
you.
C
A
Is
a
great
question:
it
does
not,
so
we
have
a
tool
as
well
as
another
Team
itself,
that
sort
of
helps
us
standardize,
namespace,
Creations,
it's
a
provisioner
tool
and
that
sort
of
integrates
with
Argo
CD
as
well,
and
the
GitHub
git
Ops
model
itself
to
create
and
manage
those
namespaces
so
that
they're
isolated
from
clients.
That's
a
great
question.
Thank
you.
A
A
Versus
multiple
yeah,
so
I
guess
the
thinking
here
was.
We
would
evaluate
that
as
we
scale
and
see
if
it's
necessary.
So
first
we
wanted
to
start
out
with
seeing
if
we
could
Shard
across
multiple
Hub
clusters,
which
is
slightly
you
know
one
step
lower
than
what
you
were
mentioning
and
see
if
that
could
actually
scale
well,
but
I
think
probably
you're,
also
bringing
up
because
of
the
security
concerns
that
they
have
on
the
security
blog
about.
A
You
know
our
back,
not
being
a
silver
bullet
and
stuff
like
that,
but
generally
I
think
when
we
were
building
this
out.
The
main
consideration
we
have
was:
how
do
we
help
consolidate
maintenance
and
best
practices
and
simplify
things
for
clients
as
while
providing
a
secure
experience
so
because
we
had
the
r
back
and
because
it
provides
secure
experience
and
it
simplified
things?
So
that's
a
great
question
if.
D
I
can
extend
just
a
little
bit.
I.
Think
one
thing
I've
seen
is
like
people
looking
at
trying
to
get
a
single
pane
of
glass
for
sort
of
deployments
across
like
an
sdlc.
If
your
setup's
not
like
that,
like
you
know,
I've
seen
setups,
where
you
have
like
a
Dev
cluster
and
product
cluster
and
the
same
versions
of
applications
deployed
to
both.
So
if
you
have
one
instance,
then
you
have
that
but
I'm
wondering
are
you
tracking
applications
like
through
a
set
of
clusters
or
are
they
kind
of
like
orthogonal.
A
So
I'm
not
sure
if
I
caught
the
entirety
of
the
first
part.
Sorry
about
that,
but
all
the
Argo
CD
applications
are
technically
living
inside
the
installation
namespace
on
the
Hub
cluster.
So
it's
it's
being
tracked
by
the
arbor
City
operator
on
the
Hub
cluster.
But
those
applications
of
course
are
deploying
onto
remote
clusters
themselves.
I'm.
We
can
also
talk
offline
because
I
feel
like
I,
might
have
missed
something
important
in
the
first
like
sentence
or
so,
and
that's
I'm,
sorry
about
that.
Yeah.
E
Yeah,
a
great
presentation,
so
I
have
a
question
like
you,
create
the
name
Space
by
different
tools.
I
think
you
can
use
a
cross
plane,
but
right
now
we
use
this
terraform
and
we
have
a
resource
quotas
on
the
top
of
namespace.
But
when
you
deploy
using
the
the
Argo
workflows
or
the
the
task,
basically
I
mean
have
you
ever
seen
like
any
problems
you
hit
the
quota's
and
all
yeah.
A
That's
that's
true.
We
have
so
that
was
part
of
why
we
had
to
modify
the
resource
allocations
out
of
the
box
for
the
workflow
executor,
so
both
for
the
main
containers,
as
well
as
the
executor
itself,
when
you
modify
the
resource
allocation
for
the
executor
that
automatically
trickles
down
to
the
init
container,
as
well
as
the
weight
containers
that
are
sometimes
affecting
whether
or
not
the
workflow
task
will
run
properly.
And
then
the
main
container
itself
is
just
where
the
task
runs.
But
that's
part
of
why
we
did
that.
A
The
namespace
level
quotas
that
we
enforce
are
configurable
so
teams
if
they
think
that
you
know
they're
running
very
heavy
workloads,
they
can
actually
actually
come
to
us
through
a
process,
a
process
in
place
for
them
to
increase
that.
But
we
have
seen
issues
and
that's
why,
with
the
workflow
level
resources
themselves
and
that's
why
we
had
to
change
those
limits
to
just
accommodate.
Okay,.
A
Well,
I
think
I
touched
on
the
provisioner
earlier.
We
just
it
just
gives
us
more
granular
control
over
how
we
manage
certain
parts
of
our
architecture,
I
think
again,
as
we
scale
up
I
think
we're
still
looking
to
grow.
We
have
a
good
multi-terrent
architecture
right
now,
we're
satisfied
with,
but
depending
on
the
needs
of
our
clients,
and,
as
you
can
see,
we
have
a
lot
of
different
use
cases.
We
might
very
well
change
based
on
where
that
direction
goes,
but
for
now
I
think
really
we
wanted
flexibility.