►
Description
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
Abusing CI/CD - Top Ways to Reach Production - Omer Gil, Cider Security
CI/CD pipelines are becoming one of the most exploited paths into internal and production systems by attackers, as there are a growing number of vectors that can be manipulated to gain access that are often times still left completely exposed, despite being well-known. From unexpected webhooks requests, to bypassing branch protection rules, through more complex attack scenarios like Poisoned Pipeline Execution - attackers have found that CI/CD is a quick way to invoke malicious attacks on production code. This talk will walk you through common attack vectors in CI/CD pipelines - ones you’re probably aware of, and ones that require more attention and research, and some of the ways you can harden your systems to prevent unwanted access to your sensitive internal data. Real-world attack scenarios will be showcased as part of the session.
A
A
So
most
of
you
probably
work
with
ccd
pipelines
on
a
daily
basis.
Nearly
any
company
that
develops
software
has
a
pipeline
to
use.
I
used
to
build
tests
and
deploy
their
code
now.
The
usage
of
ccd
is
becoming
more
and
more
common
as
years
go
by
the
best
testament
to
that
is
this
amazing
statistic
that
we're
seeing
of
the
monthly
executions
in
jenkins,
which
is
still
the
most
widely
used
build
system
in
the
world.
A
We
can
see
that
today
there
are
nearly
58
million
pipeline
execution
in
jenkins
instances
each
month
it
nearly
doubles
itself
every
two
years
kind
of
the
moore's
law
of
cacd.
Now
attackers
are
aware
of
these
statistics,
and
so
today
we
will
see
how
attackers
are
using
your
pipelines
to
reach
production.
A
A
Besides
our
product
that
we
are
highly
focused
on,
we
try
to
work
with
the
infosec
and
dev
communities
to
contribute
in
any
way.
We
can
one
example
for
that
is
the
top
10
cacd
security
risks
initiative
that
we
released
a
few
weeks
ago.
We
see
that
defenders
lack
knowledge
and
methodologies
of
how
to
protect
their
ccd
ecosystem
and,
even
more
importantly,
what
risks
they
should
protect
against.
So
this
project
aims
to
assist
us
all
in
focusing
on
threats
in
this
domain.
A
Now.
What
led
to
this
talk
today
is
the
research
that
our
team
conducted
in
the
past
year
or
so,
and
on
how
attackers
abuse
cacd
systems
today
and
I'm
here
to
share
some
of
these
insights
with
all
of
you.
So
thanks
for
joining-
and
let's
start
let's,
let's
start
with
reading
short
background,
so
with
the
rise
of
devops
and
cloud
computing,
where
everything
is
automated
and
deployed
fast
and
easy
using
code,
a
new
tech
surface
was
introduced,
which
is
reaching
the
organization's
crown
jewels
the
production
systems.
This
is
where
the
money
is
at.
A
This
is
where
the
data
is
at.
So
this
is
where
attackers
want
to
want
to
reach.
Now.
Some
of
the
most
common
attack
vectors
against
these
production
environments
could
be
first
breaching
the
perimeter
like
exploiting
a
port
open
to
the
internet
or
they're,
taking
advantage
of
a
sql
injection
vulnerability
or
any
other
vulnerability
that
you
can
take
advantage
of
in
order
to
step
into
the
systems
and
then
move
laterally
to
try
to
achieve
some
sensitive
data
or
access.
A
Now
you
might
think
you
might
think
or
ask
yourself
if
your
pipeline
is
secure,
it's
reasonable
to
say
to
think
that
it
is.
There
is
a
set
of
typical
security
controls
that
exist
to
protect
the
software
delivery
process.
You
might
identify
at
least
most
of
them.
Organizations
have
2fa
enforced
on
their
source.
Control
management
systems
or
approvals
are
required
before
merging
code
to
the
main
branch
you
might
restrict
network
access,
or
even
do
try
to
do
the
impossible
missions
of
reducing
production
permissions,
but
that's
those
controls
aren't
enough.
Now.
A
Last
year
we
witnessed
again
and
again
how
attackers
of
all
levels
of
sophistication
were
successful
in
running
malicious
code
inside
these
software
delivery
process.
One
example
was
solarwinds
where
their
ci
was
hacked
in
order
to
introduce
malicious
code
to
their
executables
used
by
their
customers
codecov.
Well,
their
infrastructure
was
hacked
to
introduce
malicious
code
to
the
script
used
by
thousands
of
customers
that
run
this
script
in
their
pipelines,
and
this
malicious
script
by
that
was
created
by
the
attackers
was
used
to
steal
environment
viables.
A
That
usually
include
the
secrets
that
were
sent
to
the
attacker's
servants
and
another
example,
which
we
see
on
a
weekly
basis.
Nearly
on
a
weekly
basis
is
compromised.
Npm
packages
like
the
popular
co-op
and
rc
packages,
usually
again
to
execute
malicious
code
in
ci
pipelines
or
in
all
in
developers
workstations.
A
And
now
you
might
think
that
your
ci
is
internal.
It's
behind
the
it's
in
the
perimeter
behind
the
firewall
protected
by
a
vpn.
So
it's
not
that
easy
to
access
and
hackers
are
like.
Oh
yeah,
challenge
accepted.
We
know
that
the
perimeter
has
long
ago
started
being
a
security
boundary.
But
more
importantly,
that-
and
this
is
also
the
focus
of
our
talk
today-
is
that
attackers
don't
have
to
have
access
or
permission
to
cacd
pipelines
to
be
able
to
run
code
inside
cacd
pipelines.
A
A
A
Now
here
comes
the
attacker.
You
can
identify
that
it's
the
attacker,
because
it's
it's
red
and
where's
the
hoodie.
Now
the
attacker
knows
that
it's
not
easy
to
directly
access
the
production
system
right
because,
as
I
just
said,
there
are
many
controls
in
place
to
prevent
him
from
doing
that.
But
he
knows
that
the
ccd,
starting
from
the
developer
and
resource
control
repository
this
eventually
leads
to
the
cxd
to
production
where
he
wants
to
to
reach.
A
So
maybe
if
we
will
get
access
to
a
developer's
account
and
get
the
right
permissions
on
a
repository
linked
with
his
pipeline,
he
might
be
able
to
go
step
by
step
until
it
reaches
production.
But
it's
not
that
easy
to
achieve
right,
because
we
see
here
just
one
developer
with
right
permissions
on
this
one
repository
he
might
be
able
to
to
fish
this
user
or
boot
for
us
their
password.
But
it's
not
really
straightforward
forward
to
compromise
a
single
developer.
A
But,
as
we
know,
this
developer
working
on
this
repository
is
probably
not
working
alone,
but
is
part
of
a
bigger
team.
So
now
we
have
five
different
engineers
that
the
attacker
can
compromise
each
of
them
in
order
to
get
these
viper
missions
that
they
need
to
access
the
depository,
but
still
getting
like
compromising
one
of
five
engineers
might
be
still
a
bit
challenging
so
lucky
for
the
attacker.
A
He
knows
that,
even
though
this
team
is
working
on
this
project,
maintaining
permissions
in
source
control
and
science
control
management
systems
is
a
huge
mess.
Probably
not
just.
This
team
has
permissions
from
the
repo,
but
many
other
people
in
the
organizations,
engineers
from
other
teams,
product
managers
qa
whatever.
A
Sometimes
in
a
medium-sized
organizations
or
even
bigger,
we
see
hundreds
or
even
thousands-
of
users
that
have
access
to
most,
if
not
all,
depositories
in
the
source,
control
management
system,
and
then
the
attacker
just
need
to
take
control
over
one
of
them
and
it
could
be
different.
Types
of
access
could
be
getting
access
to
their
password
or
ssh
keys
or
a
personal
access
token.
A
Any
type
of
access
would
be
good
in
order
to
get
these
right
conditions
on
the
repository
and
from
there
the
attacker
can
start
and
moving
between
the
different
stations
in
this
acd,
which
we're
about
to
see
now
in
order
to
reach
production.
So
now
we'll
cover
these
four
attack,
vectors
that
I
just
mentioned
that
the
attacker
can
use
to
reach
production
through
this
initial
foothold.
In
the
repository.
A
Now
the
branch
protection
was
configured
in
this
repository,
which
is
a
highly
common
security
and
quality
mechanism
to
prevent
direct
push
to
sensitive
bunches.
So
if
the
attacker
would
try
to
push
malicious
code
directly
to
the
main
branch
he
would.
This
would
fail
same
for
trying
to
push
malicious
code
in
the
jenkins
file
in
order
to
run
malicious
code
in
jenkins.
That
would
fail
for
the
attacker,
but
what
about
pull
requests
now?
A
Pull
requests,
also
trigger
pipelines
and
pull
quests
are
publicly
unreviewed
code
that
you
also
want
to
build
test
or
scan
for
security
issues
in
the
pipeline.
So
that's.
This
also
looks
like
an
interesting
opportunity
for
the
attacker,
so
the
attacker
decides
to
step
into
the
jenkins
file
code.
Let's
have
a
look
at
it,
so
this
is
the
code
of
degeneracy
that
there
are
different
stages
here,
like
the
build
stage
and
the
test
stage
and
inside
the
build
stage
that
also
run
with
the
creation
of
pull
requests.
A
We
see
that
aws
credentials
are
loaded
into
memory
as
environment
variables
and
then
some
may
commands
are
run.
What
the
attacker
decides
to
do
is
to
modify
the
junkies
file
with
malicious
code
that,
when
the
aws
credentials
are
loaded
into
memory,
he
sends
all
of
the
environment
variables
to
a
remote
server
that
he
controls.
A
So,
let's
see
what
happens
here,
the
attacker
creates
a
pull
request
with
the
malicious
jenkins
file
that
we
just
saw
now,
with
the
creation
of
the
pull
quest.
A
pipeline
on
jenkins
is
triggered
when
the
pipeline
is
triggered,
it
fetches
all
of
the
repository
code
as
part
of
the
peaceful
request,
which
contains
also
the
malicious
jenkins
file.
The
thing
is
that
this
malicious
jenkins
file
will
be
the
junkies
file
that
now
configures
the
commands
for
the
pipeline
to
one.
So
now,
malicious
code
runs
on
jenkins,
which
is
now
compromised.
A
Now
you
might
think
that,
in
order
to
execute
a
direct
fpp
attack
where
the
attacker
directly
modifies
the
configuration
files
that
defines
the
pipeline
detectors
need
to
obtain
access
to
a
developer's
account
and
if
the
attacker
already
get
access
to
a
developer's
account.
There
are
many
other
attack
vectors
to
execute
like
moving
laterally
in
the
network
and
accessing
production,
directly
etc.
A
But
I'd
like
to
remind
you
the
first,
the
first
slides
in
this
talk
that
today
many
efforts
are
being
made
to
reduce
the
attack
surface
by
other
vectors,
to
increase
prevention
and
detection
capabilities,
and,
along
with
that,
there
are
poor
capabilities
in
cacd
to
detect
and
prevent
these
kind
of
threats.
So
I'd
like
to
ask
you
if
a
direct
poison
pipeline
execution
was
executed
in
your
environment,
would
you
have
the
ability
to
know
about
it?
A
The
answer
is
probably
no
okay,
so
security
and
devops
decided
to
protect
against
the
modification
of
the
ci
configuration
files
when
running
on
review
code.
That
should
prevent
a
scenario
like
direct
ppe.
This
can
be
done
in
different
ways
like
fetching
the
jenkins
file
from
the
main
branch,
always
also
in
pull
request
or
storing
it
in
a
sensitive
repository
which
is
more
secure,
and
this
will
prevent
a
scenario
of
the
ftp,
but
does
that
actually
mean
that
your
environment
is
secure
against
pp
attacks
in
general?
So
the
answer
is,
of
course
not
otherwise.
A
A
So
here
the
attacker
starts
from
the
same
position.
It
has
white
omissions
on
the
repository,
but
now
the
jenkins
file
is
protected,
it
can't
be
modified
and
the
attacker
can't
directly
configure
what
the
pipeline
should
run,
what
command
it
should
run,
but
when
the
attacker
have
is,
has
a
look
at
the
contents,
the
other
contents
of
the
repository
he
sees
that
it
also
contains
the
make
file.
A
Let's
see
they
protect
the
junctions
file
again,
we
can
see
that
when
the
nmrs
credentials
are
loaded
into
memory,
make
build
and
make
clean.
These
commands
are
now
run
in
the
junkies
pipeline.
So
let's
have
a
look
at
the
mag
file.
We
can
see
here
the
definitions
of
the
different
make
functions
in
use
in
jenkins
and
we
can
see
a
build
plan
and
clean.
What
the
attacker
can
do
is
to
modify
one
of
these
functions
that
are
referenced
by
the
jenkins
file,
in
this
case
the
build
function
and
replace
it
with
his
own
malicious
command.
A
A
The
jenkins
file
defines
what
stages
and
steps
should
run
and
the
commands
in
in
each
step,
so
anywhere's
credentials
are
loaded
into
memory
and
then
the
mig,
the
make
build
command
are
runs,
but
now
the
make
build
command
contains
the
licious
commands
inserted
by
the
attacker
so
again,
malicious
code,
which
has
access
to
these
these
aws
credentials,
are
sent
to
the
connections
are
sent
to
the
attacker
server,
which
can
now
compromise
access
and
compromise
the
production
environment
aws
in
this
case.
So
this
is
indirect
poison
pipeline
execution
better
than
poisoning
the
pipeline.
A
Okay,
so
moving
on
to
the
third
and
last
variant
of
ppe,
which
is
a
public
poison
pipeline
execution
or
in
short,
3pe
the
3p
attack.
Now
here
the
attacker
starts
from
a
different
position
that
then
we've
just
seen
the
attacker
didn't
obtain
right
permissions
on
the
repo
instead,
in
this
case,
there's
a
public
repository
linked
with
a
jenkins
pipeline.
Now,
public
repositories,
as
you
know,
like
open
source
projects,
allow
any
users
to
contribute,
usually
by
creating
pull
requests,
suggesting
changes
to
the
code.
A
These
projects
can
be
linked
to
rci,
just
like
internal
projects
to
automatically
test
it
or
build
it.
For
example,
now
to
execute
a
ppe
attack
whether
direct
or
indirect,
the
attacker
can
fork
the
repository
and
then
it
can
modify.
You
can
modify
the
jenkins
file
to
contain
malicious
commands
similar
to
the
steps
in
direct
ppe.
A
Now,
if
the
pipeline
of
a
public
repository
is
configured
to
execute
the
pipeline
with
the
creation
of
pull
requests,
this
pipeline
in
jenkins
will
be
triggered
fetching.
The
code
form
the
code
for
default.
So
in
this
case,
and
now
this
jenkins
file,
which
was
modified
by
the
attacker
again
like
in
standard
direct
ppe,
it
will
run
the
malicious
code
created
by
the
attacker
in
jenkins.
It
will
send
the
credentials
to
the
attacker,
which
can
now
access
the
production
environment.
A
So
just
before
we
move
on
to
the
fourth
and
last
tag
vector
that
we
are
going
to
cover
today,
which
is
not
ppe,
but
something
else.
Let's
have
a
short
summary
of
comparing
ppe
to
the
a
more
traditional
active
directory
or
cloud-based
attack
techniques
to
gain
for
detectors
to
gain
a
strong
foothold
in
the
organization,
so
in
terms
of
difficulty
of
exploitation,
so
traditional
techniques
sometimes
require
more
advanced
skill
and
technical
expertise
by
the
attacker,
which
are
these
are
crafts
that
sometimes
take
years
to
learn
in
ppe.
A
You
just
need
to
create
one
pull
request
in
terms
of
likelihood
of
exploitability.
How
likely
an
environment
is
to
be
exploitable,
so
international
techniques
they
are
well
known,
and
there
are
various
protections
in
place,
as
we
already
discussed
and
in
ppe.
Nearly
any
environment
we
saw
by
today
was
susceptible
to
this
attack
for
in
terms
of
likelihood
of
detection.
A
So
again,
traditional
techniques
have
various
detections
alerts,
people
hunting
from
them
for
them
and
in
ppe,
if
you're
an
attacker
issuing
a
malicious
pull
request,
you're
unlikely
to
be
cut
and
in
terms
of
impact.
In
both
cases
the
impact
is
high.
You
in
both
of
them,
you
can
reach
sensitive
assets
like
production
environments,
so
the
choice
for
what
to
do.
If
you
gain
the
permissions
or
compromise,
the
workstation
of
an
engineer
is
pretty
obvious.
A
So
here
we
have
an
environment
where
github
specifically
is
the
sem,
and
there
is
a
ripple
linked
with
the
junkies.
Now
also,
this
environment
is
a
bit
different
that
we
than
what
we've
just
seen
in
ppe,
as
the
ci
pipeline
is
the
is
only
triggered
when
code
is
pushed
to
the
main
branch
and
not
on
creation
of
pull
requests.
Now,
branch
protection
rules
are
configured
here
as
well.
A
So
if
the
attacker
would
try
to
attempt
to
push
malicious
code
to
the
main
branch,
whether
malicious
applications,
code
or
infrastructure
code
or
an
attempt
to
execute
a
ppe
attack,
it
would
fail
so
the
possibilities
for
the
attacker
are
very
limited.
The
users
can't
approve
pull
requests
they
created.
That
would
fail
this
entire
security
and
quality
mechanisms
of
approvals
of
pull
requests.
A
So
this
is
where
github
action
steps
in,
but
why
am
I
talking
about
a
github
ci
solution
which
is
not
even
in
use
in
this
specific
organization?
So
there's
something
interesting
about
data
organization
that
you
should
know
about.
Any
github
organization
has
github
actions
installed
and
turned
on
by
default.
Any
user
that
has
right
permissions
on
the
repo
can
create
a
workflow
file.
This
is
how
actions
pipelines
are
defined
like
same
same
as
the
joints
file,
and
these
workflows
will
immediately
be
executed.
A
Sounds
like
an
interesting
avenue
for
the
attacker,
but
even
though
github
actions
won't
lead
to
directly
to
production
in
this
environment
because
it
doesn't
store
any
secret
or
have
direct
access
to
any
sensitive
asset,
but
since
something
interesting
that
the
attacker
can
can
can
try
and
use
for
for
their
purposes.
So
what
can
the
attacker
do?
A
A
The
developer
can
define
for
each
workflow
in
github
actions
if
it
would
run
with
read
or
write
permissions
against
the
repository.
Now
it
shouldn't
be
a
big
deal
right,
because
the
attacker
already
has
right
permissions
on
the
repo.
But
another
thing
that
you
should
know
about
github
actions
is
that
the
workflow
which
runs
it
runs
with
the
github
actions,
bot
user,
which
and
not
with
the
identity
of
the
developers
that
triggered
the
pipeline.
A
They
triggered
the
workflow,
so
the
bot
user
counts
as
another
developer
from
the
user
that
triggered
the
pipeline
at
the
bottom
of
the
workflow.
You
can
see
that
the
code
runs
the
code
that
runs
in
the
workflow.
You
can
select
its
simple
api
request
towards
github's
api
directing
to
approve
the
product
quest
that
triggered
this
workflow.
A
So,
let's
compile
this
whole
attack
flow
together
and
see
what
the
attacker
can
do.
So
the
attacker
creates
a
pull
request,
read
the
malicious
code
that
he
wants
to
eventually
introduce
to
production.
It
could
be
again
malicious
application
code,
malicious
infrastructure
or
even
a
direct
ppe
attack
like
modifying
the
jenkins
file.
That
is
also
in
this
repository.
A
Along
with
this
malicious
code,
the
attacker
creates
the
malicious
workflow
for
github
actions.
Now
the
attacker
carries
this
pull
request
and
then,
with
the
creation
of
this
pull
request,
the
workflow
is
triggered.
It
has
right
commissions
against
the
repository
and
it
approves
the
pr
with
the
actions
bought
the
the
github
actions
bot
user.
A
Now
I
reported
this
vulnerability
to
github
a
few
months
ago.
Through
their
bounty
program,
they
acknowledged
it
and
at
least
a
fix
that
allows
you
to
restrict
actions,
reviews
from
being
counted
towards
approval.
You
should
know
that
this
security
setting
is
not
set
securely
by
default
in
existing
organizations
just
in
new
ones.
So
if
your
github
organization
was
created
more
than
around
six
months
ago,
I
recommend
having
a
look
at
your
organization
settings
and
prevent
github
actions
from
being
counted
towards
pulling
quest
approvals.
A
So
we've
covered
four
tech
scenarios.
Let's
have
a
short
summary
of
what
we
talked
about
today.
So
we've
talked
about
three
variants
of
ppe
poison,
pipeline
execution
attack,
which
is
again
a
method
used
by
attackers
to
run
code
in
the
cr
without
having
direct
access
to
the
ci
and
from
there
move
laterally
towards
sensitive
systems.
A
We
talked
about
direct3p,
where
the
attackers
altered
the
ci
configuration
files
directly
indirect
ppe,
where
the
attackers
enter
another
file
used
by
the
ci
configuration
file,
and
we
talked
about
public,
dp
or
3pe,
where
ppe
is
exploited
through
public
repositories
and
besides
that,
we
saw
a
method
in
github
to
bypass,
to
bypass,
require
pull
request,
reviews
using
data
actions.
If,
even
if
this
ci
solution
is
not
in
use
or
was
never
used
in
your
organizations
in
your
data
organization,
so
to
summarize
pipelines,
job
pipeline
jobs
are
practically
code
execution
as
a
service.
A
This
is
how
you
see
it,
but
this
is
also
how
attackers
see
it,
and
I
would
like
to
end
this
talk
with
leaving
leaving
your
job
with
just
one
question
to
ask
yourself:
how
far
can
an
attacker
with
your
permissions
go
so
with
the
famous
words
of
torianzo
I'd
like
to
thank
you
for
being
here
today.
I
really
appreciate
it.
You're
welcome
to
the
cyber
security
blog
very
interesting
materials
that
we
keep
riding
on.
A
Cicd
security,
like
a
thorough
post
on
ppe
poison
pipeline
execution,
this
gita
vulnerability
that
we
just
covered
the
top
10
csd
security
risks
initiative,
which
I
highly
recommend
you
to
read:
the
cacd
goat
project,
the
deliberately
vulnerable
csd
environment.
In
case
you
want
to
practice
on
ccd
attacks
and
much
much
more.