►
From YouTube: Secure Multi-Tenant GitOps Application & Infrastructure Rollouts...- Vikram Sethi & Manabu McCloskey
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Secure Multi-Tenant GitOps Application & Infrastructure Rollouts At Adobe - Vikram Sethi, Adobe & Manabu McCloskey, Amazon Web Services
Securing a multi-tenant deployment for an enterprise is very challenging. Adobe built a scalable GitOps based application deployment solution for their individual teams using Argo projects. However, due to a lack of a standard solution for infrastructure automation across teams, enabling secure multi-tenant rollouts was a challenge. Adobe leveraged Crossplane in tandem with Argo to broker the provisioning of cloud resources consistently and across all teams. With this solution, Adobe and Amazon designed a layered isolation mechanism for tenant teams on top of existing shared Kubernetes clusters via a mix of technologies such as OPA Gatekeeper, ServiceAccount boundaries, IAM roles etc. This solved the non-negotiable requirements of security and multi-tenancy, which are hard to achieve natively with Crossplane and Argo. Interested? Join Adobe and Amazon engineers to hear their vision, architecture, challenges, solutions, and key takeaways.
A
Thank
you,
everyone
for
coming
for
this
session
good
morning
good
afternoon
good
evening,
depending
on
where
you're
logged
in
from,
and
it's
so
good
to
be
back
at
kubecon
in
person.
The
last
one
that
I
attended
in
person
was
that
San
Diego
in
2019.
How
many
of
you
were
there
in
San,
Diego
wow,
it's
great
good
to
see
repeat
audience
and
how
many
of
you
are
actually
aware
of
Argo
and
cross-plane
projects?
A
My
name
is
Vikram
and
I'm,
a
senior
architect
at
Adobe
I
work
in
the
developer
platforms
organization
and
these
days,
I'm
actually
working
on
introducing
Advanced
capabilities
into
our
internal
developer
platform,
and
these
capabilities
include
GitHub
space,
CI
CD.
It
also
includes
GitHub
space
infrastructure,
provisioning
and
also
building
an
AI
Ops
foundation
for
an
internal
developer
platform.
With
me
with
me,
I
have
manabu
from
AWS,
hey.
A
So
if
we
have
a
really
packed
agenda
today,
we're
going
to
start
by
talking
about
the
pain,
points,
needs
and
requirements.
Thereafter
we're
going
to
have
a
10,
000
feet,
overview
of
adobe's
services,
landscape
and
we'll
do
a
quick
overview
of
the
internet
developer
platform
and
then
Dive
Right
In
into
the
deployments,
using
our
good
projects
and
infrastructure,
provisioning
using
cross
plane
and
Argo
thereafter,
manapu
is
going
to
walk
us
through
the
multi-tenancy
and
security
requirements
and
how
we
solved
some
of
these
requirements.
A
A
So,
let's
just
quickly
have
a
look
at
the
previous
state
and
the
pain
points
associated
with
it.
So
if
you're
a
service
developer,
you
would
be
able
to
resonate
with
a
lot
of
these,
and
we
have
been
talking
to
a
lot
of
service
developers.
You
know,
as
part
of
our
you
know,
Outreach
to
the
to
the
clients,
and
we
realized
that
each
of
the
teams
are
doing
infrastructure
provisioning
in
their
own
way.
They
do
need
infrastructure
provisioning,
but
the
platform
does
not
help
them.
A
They
have
custom
tooling,
they
have
a
lot
of
learning
curve
related
to
you
know
this
infrastructure
provisioning
and
they
track
the
infrastructure
and
compute
resources
separately.
And
if
you
are
part
of
the
platform
team,
you
know
you'd
be
able
to
relate
to
some
of
these.
You
know
we.
The
platform
team
does
not
have
visibility,
observability
or
auditability
into
the
infrastructure
resources
provision,
and
we
really
have
a
hard
time
troubleshooting
some
of
this
issues
at
the
time
the
team's
face.
So
you
know
when
they
run
into
problems.
A
A
So
not
only
that
when
we
were
talking
to
the
clients,
it's
not
just
that
you
know
we
were
encountering
these
problems
in
real
life.
We
also
did
a
quick
survey.
You
know
with
the
teams
to
in
order
to
figure
out.
Okay,
you
know,
are
we,
you
know
actually
thinking
about
it
in
the
right
way,
and
actually
the
results
were
really
overwhelming.
As
you
can
see
on
the
screen
around
90
percent
of
the
developers,
they
are
asking
for
a
template
solution
from
the
platform
team
and
around
more
than
two-thirds
of
them.
A
They
wanted
the
infrastructure,
provisioning
solution
to
be
integrated
with
the
existing
github's
workflows
and
they
wanted
it
as
soon
as
possible.
So
there
was
a
customer
need
and
we
decided
to
fill
it
So
based
on
the
requirements
and
the
discussions
that
we've
had.
We
came
up
with
a
list
of
requirements
for
our
solution
that
we're
going
to
talk
about
today.
The
requirements
included.
You
know
standardization
at
Adobe
level.
Of
course
it
also.
A
We
wanted
also
the
solution
to
be
kubernetes
native
or
the
platform
itself
is
based
on
kubernetes,
and
you
know
the
new
solutions
that
we're
building.
They
are
all
you
know
the
best
order
of
githubs,
so
we
wanted
to
make
sure
that
you
know
this.
Newer
solution
is
also
gitups
friendly,
a
multi-tenancy
and
security.
As
we
talked
about
are
you
know,
key
requirements
for
for
our
platform
and
in
general,
the
focus
of
this
talk
as
well.
A
So
we're
going
to
talk
in
detail
about
these
requirements
also,
we
are
running
on
top
of
AWS
Azure
and
our
own
data
centers.
Some
multi-cloud
is
definitely
a
key
requirement
for
us
for
our
platform
and
as
much
as
we'd,
like
as
a
platform
team
to
be
able
to.
You
know,
work
on
everything
and
you
know
create
all
the
solutions
to
the
extent
possible.
A
A
So
at
a
very
high
level,
we
have
these
three
clouds,
as
most
of
you
might
be.
Knowing
we
have
document
Cloud,
we
have
Creative
Cloud
and
we
have
the
experience
Cloud.
Needless
to
say,
these
are
composed
of
various
adobe's
products
and
services.
As
you
can
see,
you
know
if
a
lot
of
you
would
know
about
Photoshop
illustrator,
InDesign,
acrobat
Lightroom,
and
you
know
so
many
other
products
that
we
have
in
our
portfolio
and
these
products
are.
A
You
know
using
a
lot
of
these
platforms
underneath
like
content
platform,
data
platform
and
Ai
and
ml
platform
that
we
have
built
over
the
years.
So
what
these
platforms
have
in
common
is
we
have
adobe's
internal
developer
platform
that
they
are
running
on
top
of
okay,
so
this
is,
you
know
on
the
foundation
piece
or
you
know
the
foundation
layer
that
everything
runs
on
top
of
which
includes
the
products
and
services
directly
on
and
also
the
platforms
and
in
turn
you
know
our
internal
developer
platform.
A
So
let's
do
a
quick
overview
of
the
developer
platform
itself.
What
you
see
over
over
here
there's
a
lot
of
boxes,
I
know,
but
these
are
actually
capabilities
of
the
internal
developer
platform,
and
some
of
these
are
actually
a
lot
of
these
capabilities
are
not
really
specific
to
Adobe
at
all.
If
you
look
at
it
and
read
through
some
of
these
capabilities,
you'd
be
able
to
relate
to
all
of
them
because
they
are
independent.
A
As
such
of
you
know,
any
Adobe
specific
stuff,
but
you
know
the
key
thing
over
here
is
that
it's
divided
into
three
different
development
phases
and
there
is
a
certain
color
coding
going
on
here
in
yellow.
We
have
the
Discover
and
create
phase
to
help
teams
to
get
started
on
the
platform
and
some
of
these
capabilities
help
in
that
way,
and
these
are
mostly
day
Zero
activities.
A
What
you
see
in
green
color
over
here
is
the
integrate
and
deploy
development
phase,
and
this
helps
teams
with
you
know
getting
the
application
up
and
running
on
top
to
the
platform,
so
you
can
think
of
these
as
the
day
one
activities.
And
finally,
we
have
something
the
the
box
is
in
blue
or
the
capabilities
in
blue
which
are
related
to
operations
and
improvements,
and
these
are
helping
teams
with
the
management
and
maintenance
of
their
application
on
the
platform,
and
you
can
think
of
these
are
day
two
activities.
A
A
So,
let's
talk
about
how
we
have
organized
our
infrastructure
in
a
hub
and
spoke
kind
of
a
model
before
we
actually
go
into
the
architecture
of
the
solutions
themselves.
When
we
talk
about
the
Hub
at
the
center,
we
have
a
hub
cluster
The
Hub
cluster
is
one
where
we
have
the
tenant,
Hub
namespaces,
you
know
and
there's
one
tenant
hub,
namespace
per
tenant
that
is
allocated,
and
this
is
where
we
are
running
our
OCD,
our
events
and
Argo
workflows,
and
alongside
that
we
have
cross
brain
installed
on
the
Hub
cluster.
A
We
also
have
Argo
rollouts
installed
on
the
remote
clusters
in
order
to
facilitate
the
in
order
to
facilitate
the
advanced
deployment
capabilities
so
alongside
being
connecting
to
the
remote
cluster
The
Hub
cluster
is
also
connected
to
GitHub
Corp,
with
all
the
code
lies
for
the
various
applications,
so
the
Hub
cluster
is
kind
of
choreographing.
The
deployments
between
git
and
the
remote
clusters.
A
So,
with
that
overview
of
the
having
spoke
model,
let's
look
at
how
we
are
solving
the
deployments
related
problems
using
the
Argo
projects,
so
we
start
with
where
we
left
off.
In
the
previous
slide,
we
have
the
corporate
GitHub.
We
have
Hub
cluster.
We
have
the
remote
clusters,
we
have
the
AWS
account.
A
This
includes
the
kubernetes
Manifest
or
the
Hem
charts,
and
also
the
Argo
manifest
in
the
client
reports,
and
then
we
have
a
shared
workflows
and
events
repository
which
contains
shared
Argo
manifests
that
can
be
referenced
by
any
other
Argo
manifest.
For
example,
the
Argo
manifests
that
are
in
the
client
repo.
They
refer
to
this
shared
workflows
and
events
in
order
to
promote
code,
reuse,
okay,
so
in
the
hub
cluster
we
have
something
called
a
provisioner
or
devices
provisioning
workflow,
and
this
is
common
for
all
applications
running
in
the
hub
cluster.
A
It
works
as
a
you
know,
kind
of
an
admin
and
the
job
of
this
provisioner
is
to
be
able
to
provision
a
lot
of
these
resources
that
are
required
in
the
hub
cluster,
which
includes
the
tenant,
Hub
namespace.
Also,
the
deployment
workflow
and
events
in
the
tenant,
Hub,
namespace
and
multiple
creating
multiple
argue,
CD
applications
in
order
to
you
know,
map
the
folders
inside
the
clan
triples
to
the
tenant,
remote
namespace,
for
example.
A
So
with
that,
let's
look
at.
You
know
how
the
end-to-end
workflow
works.
So
we
start
with
you
know
any
changes
in
the
app
code
they
trigger
the
provisioning,
workflow
and
events.
The
provisioning
workflow
will
kick
in.
It
will
do
its
provisioning,
as
we
talked
about,
for
example,
if
it
needed
it
will
create
more
rbcd
applications.
It
will
also
modify
you
know
any
anything
needed
in
the
hub
namespace
and
thereafter
it
will
invoke
the
deployment
workflow
okay.
So
once
the
deployment
workflow
is
invoked
and,
of
course,
deployment,
workflow
is
based
on
Argo
workflows.
A
So
it
is
a
series
of
steps.
The
first
or
the
second
step
might
contain.
You
know
the
steps
for
building
the
images
scanning,
the
images
and
thereafter
you
know
what
it
does
is.
It
writes
to
the
k8's
Manifest
folder
inside
the
git
repo
you
know
with,
and
it
updates
the
if
the
case
manifest
folder
with
the
gitshark
corresponding
to
the
deployment.
Now
our
one
of
the
rvcd
applications
that
was
created
by
the
provisioner,
it
is
listening
to
any
changes
in
the
case
manifest
folder
and
it
gets
triggered.
A
You
know
as
a
result,
and
it
applies
the
kubernetes
Manifest
to
the
tenant,
remote
namespace,
okay,
so
the
tenant,
remote
namespace.
You
know
any
changes
that
are
applied,
the
kubernetes
resources
they
come
up,
as
you
can
imagine,
and
you
know
they
are
able
to
access
the
resources
in
the
tenant.
Don't
AWS
account
also
worth
noting
is
that
the
argumentss
folder
is
mapped
via
Argo
CD
to
the
tenant
Hub
namespace.
What
this
means
is
that
any
changes
to
the
Argo
manifest
folder.
A
They
are
applied
to
the
tenant,
Hub
namespace
and,
for
example,
you
know
you
want
to
modify
the
deployment
workflow
itself
right.
You
know
you
have
five
steps,
you
want
to
add,
you
know
two
or
more
steps
to
the
workflow,
so
you,
you
know,
modify
the
Argo
manifest
folder.
You
know
and
commit
your
changes
there
and
the
changes
are
applied
to
the
tenant,
Hub
namespace
and
then
the
next
deployment
you'd
have
the
modified
workflow
kick
in.
A
So
everything
seems,
like
you
know,
to
be
working
great
over
here
right.
So
what
is
the
issue?
The
issue
is
that
the
resources
are
actually
pre-provision
in
the
AWS
account
right,
I
mean
we
don't
want
these
resources
to
be
pre-provisioned
or
to
be
provisioned
in
a
different
way.
We
want
this
to
be
integrated
with
the
overall
workflow.
That's
why
we're
here?
Okay,
so
let's
see
how
we
can
solve
this
problem
using
cross
plane,
but
before
we
go
into
you
know,
cross
pane.
A
A
So
everything
starts
with
the
concept
of
a
composite
resource
in
the
crosswind
world,
but
it
is
an
abstract
concept.
How
is
it
defined?
It
is
defined
by
something
we
call
as
composite
resource
definition
or
xrd
for
short
and
it's
a
cluster
scope
Resource,
as
you
can
see,
as
it's
mentioned
over
here.
So
the
platform
team
defines
this
composite
resource.
A
So,
alongside
you
know
what
the
composition
really
does,
is
it
actually
composes
multiple
managed
resources
from
the
cloud
provider,
and
in
this
case
we
are
talking
about
AWS
being
the
cloud
provider
over
here,
so
it
composes
these
resources
and
that's
how
Cross
Pin
knows.
Okay,
these
are
the
resources
that
need
to
come
up
when
the
composite
resource
is
invoked
or
instantiated.
A
The
platform
team
also
defines
something
called
a
provider
config.
What
the
provider
config
really
does
is
it
provides
access
to
the
tenant
of
AWS
account.
So
the
platform
team
will
take
the
credentials
from
the
client
teams
and
create
a
provider
config
for
them
and
then
attach
it
to
the
service,
and
that's
how
you
know
cross
brain
will
sort
of,
like
you
know,
figure
out.
Okay.
This
is
how
this
is.
A
The
AWS
account
that
I
need
to
provision
the
resources
into
so
there's
also
this
concept
of
a
composite
resource
claim,
as
you
can
see
over
here,
which
is
actually
namespace
book.
So
this
is
an
important
thing,
because
the
tenant
app
it's
actually
only
using
the
composite
resource
claim,
as
you
can
see,
on
screen
and
everything
to
the
right
is
abstracted
away
from
them.
A
Everything
to
the
right
on
in
the
platform
team
concerns
is
totally
up
to
the
platform
team,
how
they
Define
it,
but
what
the
tenant
team
does
is
to
be
able
to
author,
the
composite
resource
claims
and,
as
part
of
authoring,
the
composite
resource
claims
they
specify.
What
is
the
composition
to
use?
What
is
the
provider
config
to
use?
So
this
is
how
sort
of
Cross
Pin
figures
out.
Okay,
this
is
the
composition
that
has
been
attached
to
this
composite
to
Source
claim.
A
So,
let's
make
sure
that
these
resources,
which
are
attached
to
the
composition,
are
brought
up,
and
this
is
the
provider
config.
So
this
is
the
AWS
account
where
I
need
to
provision
the
resources,
and
once
the
resources
comes
up,
you
know
we,
the
tenant.
App
is
able
to
access
the
resources
so
with
that
overview
and
a
refresher
about
the
Cross
Pin
Concepts,
let's
look
at
you
know
the
actual
workflow
using
Cross,
Pin
and
Argo.
A
So
we
start
off
where
we,
you
know
left
off
in
the
in
the
Argo
related
workflow.
We
have
corporate
GitHub
with
the
various
folders
over
there
and
we
have
the
Hub
cluster
where
we
have
provisioning
workflow
and
events
and
deployment
workflow
and
various
rbcd
applications.
We
still
have
the
remote
clusters,
we
have
a
rollouts,
is
there
and
remote
namespaces
are
there,
but
one
thing
to
note
over
here
is
that
the
tenant
AWS
account
does
not
have
any
pre-provisioned
resources,
as
you
can
see.
A
A
So
that
is
what
the
continents
use,
and
that
is
what
they
add
to
this
crossbank
manifest
folder
and
also
there
is
a
shared
infra
resources
repository
as
you
can
see,
and
at
the
top
left,
which
is
a
wrapper
which
contains
the
composite
resource
definitions
and
the
compositions
published
by
the
platform
theme
for
the
popular.
You
know
composite
resources
that
are
in
use
in
the
organization
and
and
in
the
hub
cluster
itself.
A
A
You
know
the
provisioning
related
to
the
namespaces
or
deployment
workflows
and
all
of
that,
but
alongside
that
you
know
there
is
an
additional
job
that
it
needs
to
do
now,
which
is
to
be
able
to
look
at
what
are
the
composite
resources
that
are
being
requested
by
this
application
and
if
those
are
not
really
available
on
the
Hub
cluster,
it
installs
those
on
the
Hub
cluster.
So
we
in
a
way
dynamically
adding
those
resources
to
the
hub
cluster
rather
than
all
of
this
resources
being
there
on
the
Hub
cluster
from
the
word
goal.
A
So
once
that
happens,
the
provisioning
workflow
again
invokes
the
deployment
workflow,
and
you
know
the
rest
of
the
workflow
is,
like
you
know,
pretty
straightforward.
We
talked
about
that
in
the
previous
argot
related
Slide,
the
kubernetes
resources
they
come
up
in
the
tenant,
remote
namespace
and
you
know
everything
works
as
expected,
and
we
also
talked
about
how
the
Argo
manifest
folder
is
mapped
to
the
tenant,
Hub
namespace.
A
So
any
changes
to
the
Argo-
you
know
folder,
they
are,
you
know,
applied
to
the
tenant
hub
namespace,
for
example,
deployment
workflow
needs
to
be
modified,
but
the
additional
thing
that
is
happening
over
here
is
that
the
cross
plane
folder,
is
also
mapped
to
the
tenant
Hub
namespace.
What
this
means
is
that
you
know
any
time
a
claim
is
added
to
the
Cross
plane
folder.
It
gets
applied
to
the
tenant
Hub
namespace
and
from
there
crosswind
picks
it
up.
It
says:
I
can
process
these
resources.
Let
me
make
sense
out
of
it.
A
Let
me
make
sense
of
this
claim
and
figure
out
what
needs
to
be
done.
It
peeps
into
the
claim
figures
out
that
okay,
this
is
the
AWS
account
where
it
needs
to
provision,
and
these
are
the
resources
where
it
needs
to
that
that
need
to
be
brought
up
and
it
does
the
provisioning
and
once
the
resources
are
up,
you
know
the
tenant
containers
are
able
to
access
the
resources
and
party
time.
A
Everything
is
great
so
with
that
I
think
you
know,
this
is
how
the
end-to-end
kind
of
workflow
works,
but
there
are
also
these
multi-tenancy
and
security
requirements
that
we
need
to
go
through
and
at
a
very
high
level.
We
have
divided
into
4K
four
categories.
Let's
look
at
you
know
these
categories,
one
legitimate
access
a
person
is
supposed
to
have
access
should
have
access
to.
You
know
the
resources
at
all
times.
A
Second,
there
should
be
namespace
isolation,
so
our
platform
in
general
does
not
really
allow
a
tenant
to
create
cluster
scope
Resources
by
the
tenant
teams
themselves.
So
the
client
should
be
using
the
claims
in
our
namespace
scoped
fashion.
So
that's
something
that
we
are
trying
to
sort
of
enforce
and
also
any
Rogue
actors-
and
you
know,
as
you
can
imagine,
they
need
to
be
stopped
from
exploiting
the
system
and
finally
performance
the
existing
and
the
newer
workflows.
A
Performance
related
problems,
but
so
we
need
to
make
sure
that
those
performance
problems
are
not
really,
you
know
impacting
any
of
the
clients
or
any
of
the
workflows
that
they
have.
So
with
that
I'll
hand
it
over
to
manabu
to
walk
us
through
the
various
multi-tenancy
and
security
requirements,
one
by
one
and
go
deep
into
them.
B
All
right,
thanks
Vikram,
so
let's
go
into
all
that
details
about
these
multi-tenancy
and
security
requirements
before
we
actually
they're
going
to
you
know
details
just
the
rest
of
the
talk,
we're
just
going
to
say
that
provider
config
is
equal
to
an
AWS
account.
Now,
strictly
speaking,
that's
not
true
right.
It's
more
of
a
interest.
I
am
role
than
an
account,
but
for
the
sake
of
Simplicity
I'm,
just
gonna
say
it's
an
account.
B
Okay
first
requirement
is
about
legitimate
access,
so
tenants
they
should
be
able
to
access
whatever
resource
they
create
and
in
their
own
accounts,
only
right.
So
whenever
we
give
access
to
like
another
entity
in
WS,
you
should
always
limit
what
it
can
do.
So
in
this
case
we're
just
using
AWS
roles
policies
all
that
to
just
minimize
the
permissions
that
crossplay
needs
right.
You
don't
want
to
keep
crossbarring
permission
to
create
a
you
know:
AWS
IM
user,
or
anything
like
that
right.
B
B
This
is
a
string
that
gets
passed
through
the
tenants
and
a
tenants
specified
that
in
the
IM
row
and
make
sure
that
whenever
they
want
to
assume
in
this
world
and
that
external
ID
string
must
match
otherwise,
it
will
just
deny
that
and
then.
In
addition,
we
also
leverage
Argo
cities
R
back
to
just
prevent
tenants
from
seeing
other
tenants
resources
within
the
Argo
City
UI
as
well.
B
All
right,
so
next
requirement
is
a
namespace
isolation.
Right,
like
background
is
measuring
these
provider
config.
They
need
to
be
mapped
to
their
namespaces.
What
I
mean
by
that
is
when
someone
from
like
namespace
2,
they
should
have
access
to
provider
config
too,
but
if
they
want
to
try
to
use
provider
config
one
that
should
not
happen
right,
because
these
are
different
accounts.
B
That's
a
problem,
though
right
provider,
configs,
they're,
actually
cluster's
code.
So
it's
got
it's
becoming
a
little
difficult
to.
You
know,
combine
them
into
a
namespace.
How
do
we
solve
that?
We
use
oh
before
that,
sorry.
So
what
that
means
is
that
if
someone
from
your
namespace
stood
is
to
specify
that
provider
config
one
in
the
the
Manifest
crossbanger
is
going
to
use
that
provider
config
for
you,
even
though
you're
not
supposed
to
because
there's
no,
you
know
concept
within
Crossbay
about
you
know,
matching
these
provider
config
to
a
namespace.
B
So
to
solve
that
we
use
the
composition
process,
composition
like
vigerman,
mentioning
it's
pretty
much
just
a
template
that
includes
different
managed
resources
together
and
when
you're
gluing.
All
these
managers
together,
you
can
modify
certain
certain
Fields
right.
Actually,
you
can
modify
any
fields
in
that
in
this
managed
resources.
B
B
So
if
someone
comes
in
and
they
say,
hey
I
want
to
use
provider
config
one
the
competition
process
is
going
to
say,
hey,
no,
you
know,
that's
that's
not
where
you're
supposed
to
go.
So
it's
just
going
to
the
provider
conflict
too.
Instead
of
going
to
provide
a
config
one,
now
there's
another
problem
with
the
namespace
isolation,
so
Finance
they
should
be
able
to
use
any
managed
resources
right
so
manage
resources
in
AWS
terms.
They,
you
know,
take
response
to
AWS
Services
right,
like
S3
packets
that
have
a
DB
table,
etc,
etc.
B
B
You
know:
users
from
users
from
creating
clusters,
resource
groups,
but
at
the
same
time
this
should
be
all
be
available
so
to
solve
that
cross
planning
has
this
mechanism
right,
so
it's
a
composite
resource
and
composite
resource
claims.
Claims
are
namespaced,
so
this
is
where
application
them
use
it
right,
and
then
the
claim
is
just
pointing
at
which
composite
resource.
It's
it's
supposed
to
be
put
this
magnitude
and
then
composite
resource
manages
all
the
other
things
that
you
know.
B
Clients
actually
use
now
this
works
right,
but
the
problem
is
that
there
is
a
lot
of
resources
in
AWS.
If
you
look
at
provider,
AWS
there's
what
like
300
400,
crds
right
to
create
composite
resource
for
each
of
them
and
making
electricids
for
each
of
them
that
takes
a
while.
Do
we
need
a
way
to
automate
it?
B
Thankfully,
we
actually
didn't
have
to
do
this
thanks
to
Christopher.
He
is
a
maintainer
of
the
provider
AWS
and
he
works
for
a
company
called
Dutch
credit
bank
and
they
are
using
cross
brain
in
their
environment
and
they
pretty
much
run
into
a
very
similar
program
that
we
are
running
into,
so
they
were
actually
generously
enough
to
open
social
store,
and
you
know
give
us
the
head.
You
know
how
to
use
it
and
all
that
stuff.
So
what
this
tool
does?
B
Is
it's
just
going
to
go
into
this
crd
photos,
get
all
these
cids
and
create
a
completion
and
excise
for
them,
and
then,
when
you're,
creating
those
competitions,
excelities
and
all
that
right,
you
can
also
specify
certain
requirements
say
like
hey.
You
have
to
use
this
camskey.
You
have
to
use.
You
know
this
tag.
You
have
to
have
this
label,
you
know
all
that
stuff,
so
you
can
use
that
and
then
generate
a
composition,
acceleries
everything
and
then
check
them
into
get.
B
Now.
We
don't
want
them
to
be
available.
All
the
time
like
I,
really
don't
want
them
all
of
them
to
be
available
because,
yes,
effectively
just
doubling
the
amount
of
cids
you're
installing
in
the
cluster.
So
like
becomes
measuring,
and
this
there
is
an
Argo
workflow
to
you
know,
do
exactly
that
right.
B
So
you
have
an
application
repository
that
that
trigonously
I'll
go
workflow
if
the
certain
extrudes
or
the
clients
are
missing
just
reach
out
to
the
repository
and
then
apply
that
to
the
Cross
brain
and
then
from
there
on
workplace.
The
same
so
after
all
that
the
this
is
what
the
tenants
workflow
look
like
right.
They
all
you
have
to
do
is
they
just
have
to
say:
hey
I
want
the
earliest
cluster
and
then
just
use
a
claim,
folder
and
claim
it's
going
to
create
the
composite
resource
and
a
composite
resource.
B
Now
the
next
category
is
block
exploits
right.
So,
even
though
tenants
are
not
supposed
to
be
able
to
create
cluster
resource
right,
crosstalk
resources,
it
might
be
possible
for
them
to.
You,
know,
exploit
some
sort
of
like
weakness
or
anything
like
that.
To
kind
of
you
know,
use
this.
You
know
manage
resource
manifest.
Instead
of
going
to
the
compositions
and
process
which
you
know
again,
it's
not
going
to
work
if
you
are
just
using
monetary
results
like
this.
B
So
how
do
we
solve
this?
We
just
use
open,
Oba
right,
open
policy
agent,
gatekeeper.
If
you
know,
request
comes
in
like
this,
we
just
say:
hey.
It's
not
allowed
here
is
an
example
policy,
and
you
can
use
we're
just
saying
that
hey
it's
not
part
of
the
composition.
We
will
just
deny
you
if
this
competition
is
not
part
of
the
the
Adobe
provided,
competition
will
also
deny
it
and
then
the
final
category
is
high
performance.
B
I,
don't
know
how
many
of
you
have
used
cross
brain
in
your
environments,
but
you
know
it's
so
gross
playing
right
and
then
install
a
couple
providers.
You
might
get
something
like
this
right.
You
do
like
curb,
cut
or
get
something,
and
then
you
have
to
wait.
You
know
eight
seconds
six
seconds,
something
like
that
for
you
to
come
back.
That
kind
of
problem
is
due
to
that
large
number
of
crds
installed
in
the
cluster.
B
So
in
our
case
we
only
have
like
1000
crds,
but
that's
enough
to
cause
this
kind
of
problem
already.
So
there
has
to
be
a
way
to
you
know
not
have
this
kind
of
throttling
issues
right.
If
you
are,
you
know,
if
you're
trying
to
help
our
tenants
use
this
workflow,
you
don't
want
their
workflow
to
slow
down
because
of
these
the
kind
of
problems.
B
Luckily,
we
actually
didn't
have
to
do
anything.
I
found
appound
is
the
company
behind
Crossway
they
have
been.
They
have
been
working
very
hard
with
the
kubernetes
community
to
solve
all
these
problems
associated
with
the
large
number
of
crds.
Thanks
to
the
effort,
a
lot
of
the
clients
had
issues
being
resolved
starting
from
0.24.
A
You
thanks
Manuel
thanks
for
going
into
all
these
multi
tenancy
and
security
requirements
in
detail
with
that.
Let's
try
to
sort
of
compare
the
previous
and
the
newer
developer
experience
that
we
had.
We
talked
about
the
problems
that
the
service
teams
and
the
platform
teams
have
in
the
in
the
new
world.
A
So,
as
far
as
any
new
solution
that
we,
you
know,
come
up
with
it's
not
without
its
challenges
and
unknowns,
as
you
can
imagine,
so
we
have
our
own
set
of
challenges
and
unknowns
that
we
are
sort
of
looking
into
the
first
one
being
the
Hub
cluster,
and
you
know
the
kubernetes
performance
issues
which
were
mentioned
by
so
there's
this
one
Hub
cluster
right
now
and
we
are
trying
to
figure
out
and
test
how
much
it
can
scale.
You
know
with
Argo
and
cross
plane
and
the
multiple
crds.
A
So
we're
doing
the
testing
right
now
and
in
trying
to
figure
out
that
will
it
be
able
to
scale
to
thousands
of
services,
which
is
the
kind
of
scale
that
we
are
talking
about
at
Adobe.
Okay,
the
second
thing
is
that
we've
primarily
been
investigating
the
AWS
space
and
Azure
is
something
that
we
are
starting
to
investigate
right
now
as
to
how
much
support
does
it
have
from
a
crosswind
perspective,
we
have
been
able
to
solve
a
lot
of
problems
in
consultation
with
AWS
for
AWS
team.
A
Finally,
I
think
next,
one
is
around
the
technology.
Maturity,
both
arbo
and
cross
plane.
Are,
you
know,
incubating
projects
in
cncf,
but
you
know
they're
still,
you
know
a
road
ahead
for
them
to
be
able
to
graduate.
We
keep
on
encountering
some
problems
or
the
other
here
and
there
so
we're
trying
to
sort
of
work
with
the
maybe
the
community.
In
order
to
see
like
you
know
what
kind
of
issues
that
we
have
and
trying
to
resolve
them
on
the
way
at
Adobe.
A
Of
course,
you
know,
as
you
can
imagine,
everybody
is
doing
infrastructure
provisioning
already,
but
in
their
own
way.
So
there
is
a
lot
of
tooling
inertia.
You
know
a
lot
of
teams
are
using
terraform
for
examples
who
are
trying
to
work
with
them
as
to
like
you
know
how
cross
brain
and
terraform
kind
of
work
together,
and
you
know
what
is
the
path
forward
for
them
and
also
be
able
to
provide
an
easy
migration
path.
We
don't
want
them
to.
A
You
know
jump
into
a
land
or
in
a
river
full
of
crocodiles,
as
you
can
imagine
so
also
I
think.
Finally,
we
need
that
Community
Support
as
much
as
we'd
like
to
contribute
back
to
the
community.
We
also
need
to
make
sure
that
the
community
is
supporting
us
in
our
efforts
going
forward,
so
we're
keeping
a
tap
on
that
and
working
actively
with
the
community
with
that
manabu
and
I
would
like
to
thank
you,
for
you
know,
taking
out
the
time
is
a
QR
code.
If
you
want
to
provide
feedback.