►
Description
Attacking Argo CD with Argo CD (and then Defending) - Michael Crenshaw, Intuit
Argo CD manages Kubernetes resources, and Argo CD is itself a set of Kubernetes resources. This talk will show how a lax RBAC configuration could allow users to escalate their privileges by using Argo CD to modify Argo CD. We’ll start with a trivial attack and then incrementally restrict Argo CD RBAC and Project restrictions until no attack is possible. This talk will demonstrate the process that every Argo CD admin should follow when setting up their Argo CD RBAC and Project settings.
A
A
A
A
A
What
the
allowed
destinations
are
and
a
number
of
other
rules
just
to
restrict
and
make
sure
that
your
developers
are
only
doing
with
their
applications.
What
they're
supposed
to
be
allowed
to
do
and
then
the
very
bottom
layer
of
the
argo
cd
structure
is
the
actual
resources
that
you
deploy,
and
these
are
the
things
that
you
see
when
you
click
on
an
application
in
argo,
cd,
ui,
so
config
maps
deployments,
etc.
All
those
things
that
were
represented
as
manifests
that
are
now
deployed
and
monitored
in
your
kubernetes
cluster.
A
A
I've
listed
a
few
of
the
first
things
that
I
would
attack,
but
for
this
talk
we're
going
to
attack
the
argo
cd
rbac
config
map,
and
that
is
the
config
map
that
stores
the
csv
file,
which
describes
how
users
are
allowed
to
use
the
api,
I.e,
the
the
cli
and
the
ui
to
make
changes
in
argo
cd.
So,
by
attacking
this,
we
can
potentially
escalate
our
privileges
and
do
more
interesting
things.
A
The
rbac
config
map
matters,
because
it
is,
it
basically
governs
a
second
source
of
truth
for
your
manifest,
so
you've
got
git.
You've
got
helm,
that's
one
source
of
truth,
but
depending
on
how
you
configure
your
our
back,
it
can
become
the
gatekeeper
for
the
second
source
of
truth,
the
ui,
the
cli,
the
api
and
the
rules
are
defined
in
csv
like
this.
So
we
have
different
types
of
resources
in
this
case,
we're
restricting
the
dev
groups,
access
to
the
applications,
resource
and
I've.
A
What
that
ends
up
looking
like
in
the
ui,
and
it
would
look
the
same
if
you
use
the
cli
or
the
api,
is
when
an
admin
logs
into
argo
cd.
They
see
all
of
the
applications
and
on
the
left
side
you
see
that
bootstrapped
argo
cd
application,
but
when
a
developer
logs
in
because
they're
under
that
dev
role,
they
only
see
applications
that
are
grouped
under
the
dev
project.
A
The
attack,
if
actually
performed,
would
look
like
this
on
the
left
side.
You're
an
attacker
you've
set
up
an
application
or
suppose
an
admin
has
set
up
an
application
for
you
and
what
you're
supposed
to
be
deploying
is
those
bottom,
two
resources,
the
guestbook
ui
service
and
the
guestbook
ui
deployment.
A
But
the
nasty
thing
there
is
the
config
map,
which
is
the
top
resource.
Argo
cd,
rbac
cm
when
you
deploy
an
application,
you're
allowed
to
specify
in
your
resource
what
name
space
you
want
to
deploy
to
so
I
can
set
in
this
argo
cd,
rbac
cm.
I
want
to
deploy
this
to
the
argo
cd
namespace,
where
all
the
good
stuff
lives.
So
as
an
attacker,
I
see
the
left
side.
I've
synced
it.
My
changes
to
the
config
map
have
been
applied
on
the
right
side
as
an
admin.
A
I'm
scared
to
death,
because
I
see
that
argo
cd
is
out
of
sync
and
there's
a
warning
indicating
that
some
other
application
is
managing
the
same
resource,
and
if
I
want
to
figure
out
what
exactly
happened,
I
could
click
the
argo
cd-rbacm
resource
and
I'd
see
this
on
the
left
side.
What's
currently
deployed,
which
gives
everyone
with
a
dev
role,
full
access
to
do
anything,
that's
what
all
the
stars
mean,
and
what
I
should
have
deployed
is
on
the
right
side,
pretty
restricted
our
back.
A
I
could
hit
sync
and
then
the
attacker
could
go
back
and
hit
sync,
and
we
could
go
back
and
forth
like
that.
There's
no
way
that
I
could
completely
stop
them
at
this
point,
though,
from
just
managing
their
own,
our
back
and
as
a
matter
of
fact,
at
this
point
they
could
kick
me
out
of
the
system
and
they'd
take
over
the
whole
instance.
A
Some
organization
had
rules
on
their
projects,
preventing
users
from
deploying
namespaces,
but
one
of
their
users
added
a
namespace,
manifest
to
their
manifest
repository,
and
they
did
a
refresh
in
the
argo
cd
ui
and
they
saw
the
namespace
which
worried
them
a
little
bit,
and
then
they
clicked
the
triple
dots
and
realized.
They
were
now
allowed
to
delete
that
namespace.
A
So
the
the
bug
was
basically
that
argo
cd,
let
this
resource
into
its
internal
representation
of
what
is
being
managed
by
this
application.
Nine
days
later,
we
fixed
the
bug.
We
kicked
those
resources
out
of
that
internal
resource
tree,
and
so,
if
a
user
tried
to
sync
this
or
if
user
tried
to
use
the
api,
ui
or
cli
to
modify
or
delete
that
resource
they'd
be
prevented
from
doing
that
today.
A
But
this
attack
was
really
interesting
to
me
because
I
realized
it
started
out
as
a
fairly
simple
attack.
Okay,
oh
no!
Someone
can
delete
a
namespace,
that's
one
problem,
but
if
a
user
can
also
use
the
api
or
the
ui
to
modify
the
live,
manifest
of
something
that
they've
just
added
to
their
repository,
then
they
could
edit
the
live
manifest
of
argo
cd's
own
resources
in
the
argo
cd
namespace
and
escalate
their
privileges.
So
it
took
something
like
a
7
point.
Something
cvss
vulnerability
to
9.9,
I
believe,
is
what
we
eventually
settled
on.
A
A
So,
to
avoid
those
type
of
misconfigurations,
we
have
a
few
lines
of
defense.
First,
we
can
use
git
based
restrictions,
so,
for
example,
in
your
source
control
management
provider
like
github
git
lab,
you
can
set
some
restrictions
and
you
can
also
do
some
r
back
restrictions
in
argo
cd
to
complement
those
second
type
of
defenses.
You
can
use
argo
cd's
own
project
resource
which
we
talked
about
as
sort
of
a
list
of
rules.
A
A
So
I'm
going
to
ask
you
to
imagine
a
hypothetical
argo
cd
setup
and
we're
going
to
define
three
protections
against
attackers.
First,
our
developers
are
only
allowed
to
deploy
from
github
github
repos
that
require
admin
approval
to
push.
So
as
a
developer,
you
can't
push
straight
to
any
branch
on
your
deployment,
manifest
repo.
You
have
to
fork
it,
you
open
a
pr
and
then
an
argo
cd
admin
reviews.
A
It's
pretty
common
to
want
to
give
developers
as
much
like
broad
access
to
their
applications,
for
example,
restricted
to
the
dev
project
as
possible,
because
you
want
developers
to
be
flexible
and
be
able
to
respond
to
issues
so
we'll
start
there.
The
third
part
is
we're
going
to
configure
that
dev
project
that
devs
are
allowed
to
access
to
access
any
source
repo
and
deploy
to
any
destination,
and
some
of
you
who
are
argo
cd
admins,
are
already
like
that's
a
bad
idea.
A
So
the
first
attack
that
we
can
initiate
against
that
hypothetical
setup
is
we
can
change
the
app's
source
repo.
So
remember,
we
have
restrictions
to
prevent
users
from
just
pushing
to
that
source
repo.
However,
I
can
use
the
cli
or
even
the
ui
to
patch
this
orders,
api
application
and
just
point
it
to
a
different
repo
that
I
control
completely.
I
can
add
an
argo
cd,
for
example
the
rbac
manifest
to
that
repository,
push
it
and
now
I've
taken
over
it's
pretty
easy
to
mitigate.
A
A
Keep
in
mind
if
users
can
create
repositories
that
match
this
pattern
and
they
have
control
over
the
the
push
settings
for
that
repository.
They
can
just
bypass
this.
They
just
match
the
pattern
and
they're
in
with
whatever
sources
they
want,
so
make
sure
that,
for
example,
in
github
in
your
organization,
users
are
restricted
from
creating
arbitrary
repositories
with
arbitrary
names
and
control
settings.
A
Here's
another
way
you
could
restrict
the
r
back
for
applications
which
remember
we
had
a
star
for,
because
we
want
developers
to
have
flexible
access
now,
we're
going
to
remove
a
few
things
and
the
main
one
to
focus
on
is
update,
because
that's
what
we
were
doing,
we
were
updating
the
application
manifest
to
use
a
malicious
source.
We
just
want
to
stop
them
from
updating
it.
A
It's
not
an
invalid
source.
Instead,
it's
you're
just
not
allowed
to
update
and
they
would
see
the
same
error
in
the
user
interface
so
either
of
those
two
defenses
works
you
might
want
to
use
both
of
them
in
conjunction,
for
example,
if
you're
worried
you
can't
get
a
pattern
that
is
sufficient
to
match
and
prevent
malicious
repos,
but
either
one
should
be
able
to
work,
but
there's
still
an
attack
available.
We've
we've
done
all
this
we're
protected
against
those
attacks.
A
This
is
still
vulnerable
to
a
bootstrap
attack,
because
the
argo
cdcli
has
a
flag
option
that
I
think
kind
of
gets
overlooked.
It
is
hyphen,
hyphen,
local
and
what
this
does
when
you
do
argo
cd,
app
sync
hyphen,
hyphen
local.
It
builds
the
manifest
locally
on
your
computer,
shoots
those
manifests
up
to
argo
cd
and
that's
actually
what
gets
synced
to
the
cluster.
A
This
is
awesome
for
developers
who
just
want
to
test
something
out
real
quick.
They
make
a
quick
change
locally.
They
do
a
local
sync
up
to
the
cluster.
Look
at
it
in
the
ui
looks
great.
So
now
they
do
git
commit
fire.
It
off
put
up
a
pr.
I
like
this
tool
for
admin
type
people.
I
don't
love
this
for
just
average
developers,
because
there's
a
reason
we
like
get
ops
because
you
can
trace
what
happens
and
see
the
entire
history.
A
This
flag
is
controlled
by
the
over
the
override
our
back
action.
So
you
remember
where
we
had
applications,
comma,
star,
comma,
etc,
where
that
star
is,
is
where
the
override
application
or
the
override
action
lives
and
where
we
can
restrict
it.
So
here's
your
defense!
This
is
one
of
the
four
remaining
actions
that
are
allowed.
We
just
delete
the
override
action
when
someone
tries
to
do
a
local
sync.
This
is
what
they'll
get
permission
denied
you're
not
allowed
to
override.
A
A
The
problem
with
git
based
defenses
is
all
of
it
relies
on
argo,
cd,
admins
time
and
I
don't
know
about
y'all's
organizations,
but
a
lot
of
orgs
only
have
two
or
three
people
who
really
have
in-depth
knowledge
of
argo
cd,
if
even
that,
so
taking
those
folks
time.
Just
to
review
prs
to
make
sure
no
one's
submitting
anything
nasty,
that's
going
to
be
a
lot
of
time
into
it
as
16
000
applications.
A
If
the
team
of
eight-ish
people
who
know
argo
cd,
the
best
had
to
review
all
those
pull
requests,
we'd
never
do
anything
else,
so
we
need
something
besides
just
reviewing
prs
and
we're
going
to
do
that
with
the
app
project
crd.
So
first
we'll
start
with
an
initial
setup.
Only
two
rules.
This
time
we
don't
need
the
get
restrictions,
because
that's
going
to
take
too
much
time.
A
So
the
attack
at
this
point
is
the
simple
one
we
saw
earlier.
We
have
no
way
of
preventing
someone
from
adding
the
argo,
cd,
rbac
config
map
or
some
other
sensitive
resource
to
their
manifests,
pushing
it
to
a
repo
and
hitting
sync
and
taking
everything
over
the
way
to
stop
this
attack
using
an
app
project.
Manifest
is
actually
really
simple.
You
edit
the
destinations
field
to
prevent
people
from
deploying
to
sensitive
destinations.
A
In
this
case
I've
said
okay.
I
know
I
only
want
these
resources
going
to
the
default
namespace
if
you
have
a
very
dynamic
set
of
namespaces
that
you
want
to
allow
people
to
deploy
to
it
could
get
really
cumbersome
to
have
to
always
update
this.
So
we
do
accept
multiple
destinations
or
we
accept
a
glob
pattern
in
that
namespace
field,
so
you
could
do
star
hyphen
dev,
starting
with
2.5,
thanks
to
a
pull
request
from
blake
pedersen.
A
We
now
have
the
ability
to
add
an
exclamation
point
to
the
beginning
of
the
glob,
and
that
just
means
don't
allow
anything
that
matches
this
glob.
I
would
love
to
see
people
as
soon
as
2.5
comes
out
start,
adding
exclamation
points,
argo
cd
to
all
of
their
app
projects,
because
this
absolutely
prevents
anyone
from
managing
resources
in
that
namespace.
There's
no
way
for
them
to
take
over
your
argo
cd
instance
and
start
doing
bad
things.
A
So
this
is
what
it
looks
like
with
that
rule
in
place.
When
someone
tries
to
attack
your
cluster,
you
can
see
the
config
map
in
the
argo
cd
ui,
but
that
is
it.
If
you
hit
sync,
the
sync
will
fail
with
the
error
that
namespace
isn't
allowed
and
if
you
hit
the
three
little
dots
and
try
to
delete
or
you
go
to
edit
the
live,
manifest
all
that'll
be
prevented.
You
you
get
errors
for
each
of
those
attempts,
so
nice
and
easy
way
to
prevent
those
attacks
using
app
projects.
A
Now
I
want
to
talk
about
some
advanced
use
cases,
I'd
like
to
see
hands
how
many
people
have
used
or
are
interested
in
using
apps
of
apps
or
absolute
projects.
Okay,
a
lot
of
folks,
so
this
isn't
sort
of
the
niche
thing
that
you
know
a
more
advanced
use
case.
This
is
something
that
people
are
actually
using
a
lot
which
I
love,
but
it
is
more
difficult
to
secure
and
you'll
see
why,
for
folks
who
aren't
familiar
with
apps
of
apps,
it's
kind
of
just
what
it
sounds
like
you
have.
A
A
A
So
we
have
to
protect
that
field
and
we'll
talk
about
a
couple
different
ways
to
protect
it
in
argo,
cd,
2.4.
So
the
current
release
and
prior
the
only
way
to
secure
that
project
field
is
using
the
review
pattern.
So
you
need
to
do
everything
that
we
discussed
earlier
under
git
based
defenses.
You
need
to
make
sure
that
argo,
cd,
admins
people
who
understand
the
importance
of
that
project
field.
A
You
also
need
to
make
sure
you
disable
the
update
permissions,
because
let
me
go
back
and
show
you
what
this
would
look
like.
If
you
don't
disable
update
users
can
click
one
of
these
applications.
They
can
click
the
orders.
Api
go
to
the
live
manifest
and
just
edit
it
and
change
the
project
to
whatever
they
want.
So
in
app
apps
prior
to
2.4,
you
got
to
have
people
reviewing.
A
You
have
to
disable
the
update
permissions,
starting
with
argo
cd
2.5
there's
a
new
feature
called
applications
in
any
namespace
that
it
provides
a
model
to
allow
self-service
applications
and
restrict
applications
to
particular
projects
that
you
can
define
in
a
way.
That's
safe,
I'm
not
going
to
go
in
depth
on
that.
This
is
a
brand
new
feature
and
it
needs
a
lot
of
testing.
So
when
the
rc
comes
out,
if
you're
interested
in
this
pattern
get
in
touch
with
me,
we'd
love
you
to
test
the
rc
and
see.
A
A
There
is
currently
no
mechanism
to
prevent
users
from
one
creating
new
projects
that
allowed
them
to
do
things
they
shouldn't
be
allowed
to
do,
or
two
potentially
just
editing
existing
projects
to
allow
them
to
do
things
that
they
previously
weren't
allowed.
To
do
so,
make
sure
your
argo
cd
admins
are
reviewing
all
those
pr's
and
you've
used
all
the
git-based
restrictions.
There
are
discussions
of
making
projects
self-service,
but
those
haven't
resulted
in
a
really
awesome
proposal.
Yet
I
tried
to
write
one,
but
it's
still
a
work
in
progress.
A
A
You
got
a
couple
different
ways.
You
can
defend
one
using
restrictions
in
get
and
making
sure
that
someone
reviews
all
the
changes
and
make
sure
they're
safe
for
two.
You
can
use
the
app
project
manifest
to
restrict
people
to
only
deploy
to
places
that
you
know
are
safe
and
finally,
there
are
a
few
extra
mechanisms
to
to
defend
apps
of
apps
and
apps
and
projects.
So
that's
the
bootstrap
attack.
I
hope
people
have
found
something
useful
and
something
that
they
can
take
back
to
your
setups
and
analyze
and
see
if
you're
secure.
A
So
we
ship
argo
cd
with
a
default
project
which
is
wide
open,
and
we
do
that
because
we
want
folks
to
be
able
to
test
quickly
on
their
local
machines
and
see
if
argo
cv
is
good
for
them.
I
mentioned
in
the
workshop
yesterday
someone
wrote
an
article
dennis
anastasio
about
how
you
should
restrict
that
default
project
down.
A
The
first
thing
you
should
do
is
try
to
limit
it
down
to
only
what
you
need,
or
even
just
completely,
restrict
it
and
then
use
new
projects
to
restrict
out
the
argo
cd
namespace
by
default
is
something
I
would
want
to
look
at
for
3.0.
I
think
that
that's
probably
a
good
starting
point.
It
would
make
onboarding
more
difficult
for
people,
but.
A
A
Understandable,
thank
you.
I
hope
it
was.
I
hope
it
was
clear,
other
questions
and
just
to
expand
on
that.
I
think
there
are
a
number
of
things
that,
in
like
a
major
release,
3.0
we'd
want
to
change
defaults,
maybe
in
ways
that
slow
folks
down
a
little
bit
but
would
help
people
you
know,
take
care
of
their
setups
a
little
bit
more
easily.
A
C
C
What
what's
the
relationship
of
the
ecosystem
of
different
vendors
creating
their
own,
because
I
was
exposed
before
with
starboard
from
aqua,
and
I
always
felt
like
they
were
kind
of
doing
this
in
a
canonical
way.
But
now
that
I'm
seeing
the
bigger
picture
in
devops
and
see
different
vendors
creating
their
own
things.
So
for.
B
C
A
A
A
A
I
think
that
someone
thinking
about
argo
cd
in
its
relationship
to
basically
how
kubernetes
asks
people
to
secure
things
is
probably
a
good
way
to
to
think
about
it
as
far
as
just
in
general,
good
ways
to
understand
security
with
with
ci
cd
tech,
a
lot
of
it's
very,
very
new
stuff
and
very,
very
quickly,
evolving
stuff.
I
highly
recommend
thinking
about
how
you
would
attack
your
cluster.
A
C
A
A
A
Oprah
rules,
okay,
so
the
question
is:
have
we
explored
integrating
oprah
rules
for
our
back?
The
last
part,
I'm
not
sure.
I
understand,
though,
oh
for
argo
users,
kind
of
so
alex
from
acuity
has
an
issue
up
and
I
think
it's
in
the
2.6
milestone,
but
it's
basically
a
field
and
app
project.
That
is
a
jq
query
and
any
resource
that
you
attempt
to
to
apply.
That's
under
this
project
that
jq
query
is
going
to
run
and
if
it
returns
a
truthy
result,
it's
allowed
if
it
returns
a
false
result.
It's
not
allowed.