►
Description
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
Decoupling DevSecOps from CI/CD Pipelines - Kayra Otaner & Jeff Woods, ADP
Having a monolithic CI/CD pipeline for CI & CD and Security (SAST/DAST/SCA) is very common, however maintaining single CI/CD to take care of all security needs for app dev is becoming an anti-pattern. Security slows down CI & CD, and creating friction between Sec * Dev needs. Creating completely separate/shadow/parallel universal pipeline for DevSecOps needs seems like working for enterprises like ADP.
A
A
If
you
have
players
over
here
for
you
folks,
it's
typically
very
hard
to
talk
about
the
you
know:
Security
in
general,
I'm,
just
gonna
talk
about
their
high
level,
ten
thousand
foot
over
what
we
have
been
doing
at
by
show
of
hands.
Can
you
guys
show
if
you
have
any
security
integrated
with
your
Pipelines
before
we
start
so
very
few
people?
A
Do
you
guys
operate
like
regulatory,
complex
or
under
Regulatory
Compliance
requirements
like
HIPAA,
PCI,
sucks,
sorry,
very
few
people,
so
you
guys
are
not
mandated
to
integrate
your
security
into
your
infrastructure
all
right.
So
the
the
title
is
decoupling
devs
that
comes
from
cicd
Pipelines.
We
are
with
ADP.
We
have
couple
slides
to
initialize.
Also
10
of
them
are,
you
know,
gonna
be
presented
by
me.
The
rest
are
going
to
be
present
by
Jeff
Woods,
my
partner
in
crime.
A
I'm,
the
director
of
devs
accounts
with
ADP
and
Jeff
is
principal
architect
with
ADP.
We
are
devseconds
at
adp.com,
so
disclaimer
openings
Express
in
this
presentation
are
our
own
and
not
necessarily
those
of
our
employers.
Just
so
you
guys
know
so
very
quickly.
I
have
in
this
space
since
1994
Linux
from
scratch,
user
have
been
delivering
speeches
talks
in
a
lot
of
Eastern
Europe
in
most
countries,
Russia
China,
turkey
and
states
have
been
over
here
for
20
years.
A
B
Yeah
spent
most
of
my
career
working
as
a
developer
disclaimer,
you
know
I
do
still
identify
as
a
as
a
developer.
I
might
say
some
mean
things
today
about
development,
but
it's
all
in
in
improving
the
the
practice
right
about
five
years
ago,
I
started
moving
into
the
security
domain
right
because
I
realized
that
that
security
was
more
important
than
most
developers
give
it
treatment
for
contact
information
up
on
the
on
the
board
there.
So
I'm,
not
gonna
rate
it
all.
B
A
Since
we've
been
in
this
space
for
a
very
long
period
of
time,
we
have
come
to
realize
that
there
are
anti-patterns
Brewing
right
under
Plain
Sight
inside
the
umbrella
of
devops.
They
have
stack
UPS.
So
we,
the
title
of
this
presentation,
is,
as
you
know,
is
decoupling
devsecops
from
cicd
and
what
you
might
say.
Why
are
we
decoupling
devs
accounts,
which
everyone
is
trying
to
put
together
automation
around
security
or
security
infrastructure?
So
let's
take
a
look
at
the
couple
good
examples
of
what
anti-patterns
is
and
let's
take
a
look
at.
A
The
definition
of
what
anti-pattern
is
anti-pattern
is
a
common
response
to
a
recurring
problem
that
is
usually
ineffective
and
highly
risk
risky
risk
being
highly
counterproductive
right.
Just
going
to
look
at
a
couple
of
those
examples
that
we
have
come
to
realize
and
understand,
you
know:
I,
look
by
looking
at
the
current
interpretation
of
what
devsecops
pipelines
are,
so
this
very,
very
famous
pipeline
or
the
infinity
loop
that
shows
you.
This
has
been
around
for
almost
like
10
years.
A
By
now,
what
devops
looks
like
so
devops
pipeline
or
SDSU
lifecycle
governed
by
the
DeVos
pattern
is
typically
a
sequential
steps
that
starts
with
decoding
building
testing,
releasing
deploying
operating
and
monitoring
and
dump
starts
the
loop
again,
that's
a
kind
of
classical
infinity
loop.
That
shows
you
a
waterfall
of
pipeline,
so
you
might
have
parallel
steps,
multi-threaded
or
parallel
extension
of
those
steps
in
a
single
pipeline.
But
that's
typically,
you
know
people
typically
operate
a
single
pipeline
multi-pipeline.
That
does
everything
for
you
right.
So
this
was
2000
that
was
2012.
A
so
in
in
around
2018
2019.
The
devsecups
or
secops
patterns
started
to
emerge,
so
big
Enterprises
started
integrating
gently
integrating
security
into
their
pipelines
again
a
single
pipeline,
but
in
the
middle
you
have
security
this
from
Gartner.
You
have
this
in
inner
Loops
that
you
might
see
over
here,
monitoring
analytics
inside
and
there
are
some
security
related
steps
going
around
it.
But
security
is
still
like
kind
of
outside
the
perimeter
kind
of
outside
that
that
pipeline,
but
trying
to
integrate
that
with
the
devops
or
release
pipeline.
This
version
2019-
and
here
comes
2022.
A
this
from
Carnegie
Mellon
University.
They
do
organized
devsec
upstates,
they
are
a
big
DOD,
a
shop,
so
they
have
been
developing
this
new
modern
devsecops
model
or
Infinity
Loops,
which
you
might
see.
There
are
a
lot
of
inner
circles
over
here
that
you
have
to
operate
a
Dev
cycle,
a
loop
and
upcycle,
and
you
have
sec
and
it's
complicated.
You
have
SAS
threat,
modeling,
secure
coding,
secure,
config,
secret
transfer,
50
other
things,
so
security
is
not
that
easy
right.
Security
at
its
heart
is
very
complicated.
A
It
requires
specialization
on
certain
area
integrating
that
security
into
a
CI.
Cd
pipeline
has
been
a
challenge
and
we
came
to
realize
that
this
is
becoming
a
bigger
problem,
so
the
big
Enterprise
started
decoupling,
that
from
the
from
the
primary
line.
So
when
you
Google
or
like
search
for
devsecups
or
devops
patterns,
you
come
to
realize
that
a
lot
of
to
most
startups,
they
do
block
a
lot
a
lot
of
about
about
a
lot
of
the
cicd
experimentation
that
they
have
been
going
through.
A
They
come
up
with
good
blueprints
that
works
for
for
their
scale,
but
if
you're
under
operating
under
a
Regulatory
Compliance
or
if
you're
inside
a
big
Enterprise,
a
lot
of
you
guys
must
most
like
they're,
going
to
be
becoming
or
either
working
for
a
big
Enterprise
or
like
trying
to
become
a
big
Enterprise
because
startups
are
very,
you
know,
ambitious
about
that.
The
one
size
fits
all
approach
does
not
work
for
CI
CD,
that's
what
we
call
one
size
fits
none
happen
happening
all
across
it.
You
know
organizes
organizations.
A
So
one
thing
that
is
very
easy
to
identify.
You
know
is
we
we
start
naming
it
as
DIY
effects,
so
how
many
of
you
guys
are
creating
your
Docker
files
come
on
almost
like
the
90
of
you
guys
are
creating
Docker
files.
So
it's
very
easy.
You
can
watch
a
YouTube
video
that
shows
you
how
to
create
a
Docker
file
or
create
a
container
image
right.
It's
a
copy
paste
most
of
the
times.
You
can
go
to
stack,
Overflow
and
copy
paste,
something
that's
over
over.
That
has
been
there
already.
A
You
know
for
for
a
lot
of
other
people
as
well,
so
I
have
been
going
through
my
home
remodeling
thing
and
I
have
been
watching
a
lot
of
YouTube
videos
on
internet
for
DIY
Stuff
how
to
put
the
crown
molding
how
to
do
the
remodeling
in
your
kitchen
and
whatnot.
It
seems
very
easy
right,
but
once
you
start
creating
your
own
stuff,
once
you
start,
you
know
working
on
desktop.
You
come
to
realize
that
there
are
a
lot
of
steps
that
can
go
wrong.
A
Palo
Alto
has
a
unit
called
Unit
42..
These
guys
are
focusing
on
a
lot
of
attacks.
That's
happening
or
responding
to
attacks.
99
of
the
kubernetes
helm.
Charts
have
insecure
conflicts,
public
usable
ham,
charts,
it's
very
easy
to
create
a
Hampshire
kubernetes
manifests
are
right,
very
easy.
91
percent
of
the
images
on
public
registers,
you
name
it
have
critical
findings.
A
If
security
is
very
easy
right,
it
should
be
very
you
know.
Naturally,
you
know
we
should
have
at
least
51
percent
of
those
container
images
created
in
a
proper
manner
right.
That's
not
what
we
have
been
saying
if
it
seems
easy,
most
likely,
you're,
just
gonna
be
overseeing
something
over
the
simplifying
stuff
through
it
is
really
pure
and
never
simple.
Right.
A
Security
is
not
that
easy
to
integrate
with
your
primary
pipeline,
so
asking
developers
to
integrate
under
the
umbrella
term
called
we
called
shift
left
asking
developers
to
integrate
security
into
their
infrastructure
is
a
mistake,
so
you
have
to
have
security.
People
get
more
involved
with
development.
That's
what
actual
shift
life
should
should
look
like.
So
most
people
when
we
say
when
we
talk
about
hey,
we
are
operating
devs
that
comes
infrastructure
developers
are
primarily
asking
hey.
Are
we
going
to
be
able
to
deploy
stuff
or
have
root
access
to
production?
A
A
If
you
look
at
the
conventional
software
development
life
cycle,
you
you
will
end
up,
seeing
like
your
code
is
either
sitting
in
the
source
stage.
That
is,
you
know,
plain
text
sitting
in
git
or
bitbucket
reposted
or
whatever,
and
then
you
build
that
into
an
artifacts
that
artifact
becomes
your
binary.
It
can
be
content,
image
or
jar
file
or
War
file
that
you
can
deploy
to
production
and
you
have
the
delivery.
Stitch
right.
Delivery
stage
is
like
you
deploy
to
your
QA
environment,
Dev,
environment
problem
or
whatever.
A
So
we
have
continuous
integration
pipeline,
the
sci
and
we
have
continuous
derivative
pipeline
that
spans
all
of
those
stages.
So
continuous
integration
pipeline
brings
your
source
code
into
an
artifacts
State
and
over
here.
When
you
have
that
artifact
build,
you
can
do
SDA,
Source
composition,
analysis,
which
you
know
Art
Factory,
jfrog
users
have
been
doing
already,
but
if
you're
in
the
source
stage,
you
can
still
do
the
s-bomb
generation
or
static
application,
you're
testing
in
order
for
your
Dynamic
application,
secure
testing,
which
is
primary
things.
A
What
for
for
people
who
are
operating
devsecups
infra
is
to
get
your
code
into
the
delivery
stage
or
application
compiled
and
deploy
somewhere.
So
you
can't
do
the
desk
testing,
which
you
know
we
use
zap.
We
use
verb
sheet.
All
other
tools
out
there,
but
you
have
to
get
that
to
the
through
the
pipeline,
so
these
are
typically
executed
by
the
same
individual
development
teams
that
primary
owns
that
build
steps,
integration,
steps
and
continues
their
pipelines,
but
integrating
SAS
and
Das
has
been
a
challenge.
A
So
we
came
to
realize
that
you
know
deploying
at
a
deep
dedicated
part
of
that
pipeline,
just
focusing
on
the
source
stage.
We
call
the
security
escort
Pipeline
and
not
as
the
develop
development
teams
to
own
it
owned
by
security.
Folks
only
just
to
create
you
know,
SAS
findings
or
s-bomb.
That
has
been
very
efficient,
so
you
can
create
a
completely
decoupled
pipeline
that
does
security
s
code
and
looking
in
the
source
stage,
without
even
talking
to
those
development
teams
completely
decoupled.
So
by
the
advances
they
might
not.
A
You
might
not
know
about
these,
but
there
is
CNB
is
cloud
native,
build
packs
or
Jeep
or
co.
These
tools
gives
you
ability
to
convert
a
source
code
to
a
container
image.
You
don't
need
a
Docker
file.
That's
why
I
asked
if
you
guys
are
have
been
writing
Docker
files
I
can,
by
reading
a
source
code
reposter
without
a
Docker
file
existing
in
that
source
code
repository,
convert
that
reposter
into
a
container
image.
A
If
that
reposter
is
in
a
buildable
state,
we
don't
need
Docker
files,
so
the
end
result
is
going
to
be
container
image
by
having
that
policy
as
code
pipeline,
which
we
have
been
focusing
on
for
a
while,
we
can
get
the
source
code
independent
of
all
the
CI
CD
pipelines.
Get
that
converter
container
image.
We
can
still
do
the
sca
source
code
analysis
and
get
that
deployed
to
a
dash
from
the
cluster.
If
it's
a
monitor,
it's
very
easy.
A
One
final
anti
pattern
thing
that
we
have
been
observing
you
guys.
This
is
a
very
famous
blog
post
from
V
works
and
I
love
their
stuff.
You
guys
know
the
push
versus
pull
model
right
common
model
is
that
you
have
a
service
that
is
doing
your
response
for
your
cicd
stuff,
say
it's
a
Jenkins
server
that
that
has
production
credentials
that
that
lasts
it
to
deploy
your
artifacts
content
enemies
or
hamsters
whatever
to
your
production
infra.
A
So
you
have
to
add
that
secret
access
credential
stored
in
your
Jenkins
to
push
your
deployments.
This
is
called
push
model.
The
pull
model
is
you
have
an
agent,
it
can
be
text
on.
It
can
be
Argo
CD.
It
can
be
anything
else
that
has
been
running
inside
your
operator
inside
your
kubernetes
control
plane.
That
has
the
credentials
to
your
search,
control,
repository
or
content
registry
that
pulls
the
code.
The
entire
pattern
here
is
that
everyone
assumes
that
the
right
way
to
do
this
stuff
is
to
have
your
Jenkins
server
to
push
the
application.
A
That
should
not
be
the
case
if
you
are
on
the
regulator,
compliance,
because
your
production
credentials
are
going
to
be
all
over
the
place
if
you
give
them
to
your
junker
server
or
any
other
CSD
server.
So
we
have
seen
coupled
patterns
over
here
anti-patterns
over
here.
Pull
model
is
very
important.
You
have
to
get
your
production
credentials
locked
into
your
production
Zone
and
not
leave
anywhere,
not
live
anywhere
else.
The
second
one
is
security
escort
SAS
task
cable.
A
This
can
be
decoupled
from
the
rest
of
the
pipelines
very
quickly
by
the
advances
of
like
cmbs
and
other
tools
and
our
policies
code.
Like
Opa
policies,
you
can
also
get
them
deployed
independent
of
the
cicd
steps,
that's
primarily
very
ten
thousand
foot
overview
of
what
we
have
been
working
on,
I'm
just
going
to
give
the
stage
to
Jeff
so
that
he
can
go
over
a
couple.
Other
slides,
okay,
uh-huh,
it
doesn't
show
you
yeah.
B
B
You
have
centralized
expertise
on
configuration
management,
deduplication
of
of
the
findings
that
are
coming
out
of
that
tool
and
you
have
consistent
results
coming
out
of
the
tool
right
there
we'll
touch
on
this
again
in
a
minute,
but
somebody
in
your
organization
is
going
to
want
to
know
that
security
is
being
done,
especially
if
you've
got
any
kind
of
compliance
requirements.
Okay,.
B
B
How
many
how
many
developers
are
going
to
know
the
difference
between
a
timing
based
SQL
injection
attack
or
a
a
stored
or
reflected
SQL
injection?
You
know
some
of
these.
Some
of
these
attacks
get
pretty
deep,
require
a
lot
of
expertise
to
understand,
and
these
are
the
people
that
you're
asking
to
select
your
security
tools
right
as
sonar
cube.
A
security
tool
is
check,
marks,
coverity
a
suitable
security
tool
right
who's,
doing
the
compare
and
contrast
the
feature
evaluations.
B
B
Okay,
many
organizations
simply
allow
their
developers
to
to
pull
the
the
open
source
projects
that
they
want
to
use.
Okay,
bad
idea
for
a
number
of
ID
for
for
a
number
of
reasons
here,
developers
want
quick,
wins,
that's
their
motivations,
okay;
they
they
want
something
that
they
can
plug
in
and
move
on
to
the
next
feature,
and
they
can
they
can
deliver
the
product
as
quickly
as
possible.
B
Problem
is,
is
they're
they're,
not
thinking
about
the
impacts
of
of
Licensing
or
the
impacts
of
features,
whether
they're
being
used
or
not
in
that
library
or
or
or
product
they're.
Not
thinking
about
the
ongoing
concerns
that
a
security
team
may
have
such
as
are
the
proper
mailing
lists
being
monitored
for,
for
instance,
a
show
of
hands
just
who's
who's
using
Jenkins
right
I
would
imagine
it's
a
large
portion
of
the
crowd.
How
many
of
you
know
what
version
of
Jenkins
you're
running
on
your
production
clusters?
B
B
A
I
don't
want
to
interrupt,
but
you
know
security
is
not
that
easy
right
and
I'm
just
going
to
give
you
a
good
example.
If
you
have
noticed
someone
was
presenting
on
this
stage
like
20
minutes
ago,
with
the
Chrome
showing
screaming
for
I
need
an
update
red
update
on
the
upper
right
and
I.
All
these
percentages
that
I've
been
attending
as
a
speaker
or
like
as
an
audience
like
at
least
20
30
of
the
chrome
chrome
browsers,
are
screaming
for
an
update
how
hard
it
is
to
click
that
update
button.
A
It
is
not
that
hard,
but
we
keep
we
tend
to
ignore
it.
The
main
thing:
maintaining
that
update
is
a
challenge.
If
you
are
not
OCD
of
such
a
compulsive
disorder
on
that
space,
you're
not
going
to
be
able
to
you
know,
update
it.
I'm
not
complaining,
I'm,
not
saying
that,
there's
a
mistake,
but
that's
the
typical
human
behavior,
that's
the
anti-pattern.
They
have
to
have
a
dedicated
team
to
update
those
stuff
for
you.
B
Or
at
least
to
raise
the
issue
so
that
it
can
be
updated
right,
I
mean
as
a
security
team.
We
are
not
going
to
go
in
and
update
your
libraries
for
you.
You
know
if
you're
on
development,
but
it
definitely
needs
to
be
pointed
out
right
shift
left
is,
is
another
one
of
those
things
that
we
we
hear
right
and
all
this
is
doing
is
moving
security
earlier
into
the
into
the
pipeline.
B
Not
that
that's
a
bad
thing
by
any
stretch
of
the
imagination,
okay,
but
problem
there
is,
you
know
to
to
do
security
properly.
It
can
take
a
long
time
right,
I
mean
we've.
I
know:
we've
got
some
some
static
code
analysis
that
that
runs
measured
in
days
right.
We've
got
many
that
that
are
measured
in
hours.
This
is
not
something
that
a
development
team
is
going
to
tolerate,
having
run
as
part
of
their
CI
pipeline.
Possibly
you
know,
merging
from
Dev
to
to
uat
or
to
to
release
or
to
master.
B
B
I
have
personally
worked
on
development
teams,
where
I
have
had
to
go
to
the
the
compliance
organizations
and
certify
that
what
we're
doing
maps
to
various
control
standards
at
least
twice
a
year
right
and
and
these
engagements
often
run
you
know
a
week
or
two
at
a
time
and
would
pull
me
away
as
an
architect
from
the
from
the
team
to
defend
what
we
were
doing
to
ensure
that
the
the
software
was
being
built
responsibly
right.
So
kind
of
the
conclusion
that
we've
come
to
is
decoupling.
B
B
B
A
Yeah,
this
might
sound,
like
you
know,
anti
devops
or
devsecops,
because
the
core
promise
of
devops
there's
breaking
up
silos
right.
You
end
up.
You
try
to
merge
or
melt
Dev,
apps
and
security
into
one
group
of
people,
but
there
are
certain
specializations
that
needs
to
happen.
Something
that
seems
very
easy,
simple
to
implement
doesn't
mean
that
you
should
be
implementing
it
and
we
have
shown
those
examples.
A
You
know
I
don't
want
to
give
examples
of
anyone,
but
you
know
a
lot
of
corporate.
You
know.
Enterprises
have
been
running
containers
as
Docker
running
as
roots.
A
lot
of
people
are
have
been
adding
on
understand
number
of
dependencies
to
their
container
images.
So
these
are
not
the
best
practices
that
you
should
be
following.
A
So
if
you
ask
developers
to
create
a
Docker
file,
which
is
the
modern
way
of
shipping
applications,
if
you
are
asking
them
to
own
the
security
applications
connected
stuff,
which
is
very
simple,
you
just
watch
a
YouTube
video
and
get
them
implemented.
You
are
doing
the
right,
you're
not
doing
the
right
thing,
because
asking
developers
to
own
that
aspect
of
the
operation
is
not
the
right
approach.
A
You
should
ask
security,
conscious
people,
you
should
have
been
the
team
they've
been
pushing
forward
for
for
a
while,
as
security
folks
get
to
get
more
involved
with
developments.
That's
why
you
know
you
know.
Decoupling
security
has
to
take
place
right,
do
not
try
to
integrate
all
those
SAS
SCA.
Numerous
number
of
other
steps.
There
are
security
into
the
CI
CD
pipeline
split
them,
give
them
to
a
separate
individual
team
that
has
more
specialization
in
that
area,
not
because
it's
hard,
but
it
needs
specialized
eye
on
that.
A
On
that
thing,
and
as
you
might
have
seen
this
s-bombs,
you
know
the
solarwinds,
you
know
all
these
Ubers,
you
know
log
4J,
Apache,
you
know
text
four,
shell,
you
know
spring
Force.
Shell
have
been
accelerating
the
increase,
the
risk
of
getting
hit
by
that
type
of
a
vulnerabilities
increase.
Only
increasing
you
cannot
have
enough
dedicated
development
resource
developer
resources.
A
A
So
every
day
changes
you
give
your
CV
on
that
it's
almost
like
weather.
It
shows
you
between
zero.
One
shows
a
49
percent
probability
probability
of
that
getting
exploited,
focus
on
those
epss
score.
Higher
IEPs
score
once
do
not
try
to
fix
anything,
there's
also
another
one
which
is
from
sisa
sisa.gov.
A
A
They
start
walking
in
the
you
know
the
Sahara
like,
and
you
know
they
have
their
running
shoes
and
everything
back
in
their
backpack,
and
they
see
cheetah
coming
approaching
them
from
the
distance
and
one
of
the
guys
starts
running,
which
is
like
Red,
Bull,
commercial,
I,
guess,
one
of
the
guys
start
running
and
the
other
guy
starts
removing
his
backpack.
Putting
his
running
shoes
on
and
the
guy
who
started
running
first
says:
oh,
are
you?
A
Do
you
think
that
you're
going
to
be
running
faster
than
cheetah,
which
is
the
fastest
animal
on
earth
and
the
other
guy
who's?
Putting
his
shoes
on
says
I,
don't
have
to
run
faster
than
him.
I
need
to
run
faster
than
you
right,
so
you
don't
do
not
try
to
patch
everything.
That's
the
antibond!
That's
your
urge
that
you
want
to
patch
stuff
very
quickly.
This
is
not
going
to
work
you're,
not
just
going
to
lose
burn
a
lot
of
creditors
and
not
do
anything.
Meaningful
use.
Epss
use
a
vulnerable,
exploit
a
vulnerabilities.