►
From YouTube: Everything You Should Be Doing, But Aren’t: DevSecOps for K8s Workflows - Steven Terrana
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Everything You Should Be Doing, But Aren’t: DevSecOps for K8s Workflows - Steven Terrana, Booz Allen Hamilton & Dan (POP) Papandrea, Sysdig
Steven and Pop will describe a defense-in-depth approach to secure production workloads running on Kubernetes. We’ll show a live demonstration of using CNCF projects like Helm, OPA, Falco, and Argo to secure Kubernetes clusters. With a secure cluster, they will then show you how to leverage DevSecOps principles to incorporate security into every step of the software development lifecycle and how to scale your CI/CD pipelines using the open source Jenkins Template Engine. This talk is the result of lessons learned supporting multiple horizontals of end users, including FinTech and modern Federal software delivery. Specifically, attendees will walk away with actionable plans for how to implement: - Application Security - Configuration Management Policies - Runtime Threat Detection - Governance as Code - Post-mortem Forensics
https://sched.co/ekFo
A
A
B
There's
there's
a
lot
to
do
when
it
comes
to
securing
your
application,
development
and
actually
implementing
devsecops.
So
this
is
our
our
entire
talk
in
a
slide
right.
So
the
first
thing
is:
what
is
devsecops?
I've
had
the
pleasure
to
to
talk
to
hundreds
of
people
about
that
question
and
it
turns
out
no
one
actually
agrees.
So
here's
a
slide
that
tries
to
level
set
what
exactly
we
mean
when
we,
when
we
say
devsecops,
the
shortest
definition
I
can
come
up
with
is
integrating
security
into
every
step
of
the
software
development
lifecycle.
B
What
does
that
actually
mean
right?
So
you've
got
application
dependency
scanning
static
code,
analysis,
container
image,
scanning,
continuous
compliance,
dynamic
application,
security,
testing,
accessibility
assurance
all
of
those
things
work
together
to
build
a
trusted
software
supply
chain.
I'm
going
to
dive
into
more
detail
about
what
each
of
those
things
actually
means.
B
The
end
of
that
is
is
a
container
image
we're
at
kubecon
that
gets
deployed
onto
a
cluster.
Then
there's
still
more
to
do
right,
you,
the
all.
The
security
in
the
world
is
not
going
to
help
you
if
you're
running
your
containers
as
the
root
user
and
privileged
and
new
cves
come
out
every
day.
So
you
need
to
do
continuous,
runtime
security
and
we're
going
to
walk
through
how
to
actually
accomplish
that.
A
And
one
thing
that
you
know
to
note
here:
stephen
is
and
for
everybody
out
there
if
you're
watching
you
don't
get
these
things
out
of
the
box.
These
are
not
something
inherently,
you
know
possible
with
you
know,
out
of
the
box
with
kubernetes,
so
big
big
thing
out
there
for
the
101
out
there
who
haven't
deployed
a
kubernetes
cluster.
B
No,
we
just
have
so
much
synergy
that
it
pains
me
so
how
to
build
a
pipeline.
If
you're
like
me,
you
you
learn
through
stack
overflow
in
google
and
what
you're
going
to
learn
on
google
is
open.
Your
ide
write
like
five
lines
of
code
from
a
tutorial
that
gets
you
to
step
two
here
which
are
automate:
how
to
run
some
unit
tests
in
a
pull
request.
B
B
B
The
downside
is
that
each
of
those
microservices
needs
a
pipeline
right
and
every
ci
cd
tool
in
the
industry
is
focused
on
building
a
pipeline
for
a
single
application,
and
what
we
realized
was
that
different
types
of
applications
use
different
tools,
but
process
really
changes
so
usually
you'll
write
your
700
line,
jenkins
file,
that
does
all
the
different
types
of
quality
testing
from
unit
testing
to
integration,
testing,
browser-based,
test
automation,
pull
a
word
out
of
the
dictionary
and
put
testing
on
the
end
of
it.
Then
you
throw
in
all
your
security
scanning
and
you're.
B
B
B
It
took
you
like
three
seconds
to
draw
700
lines
to
write
for
one
app
and
then
you
copied
and
pasted
it
a
hundred
times
for
each
microservice
and
tweaked
it,
depending
on
the
types
of
tools
that
are
being
used
right.
So
the
reason
that
this
is
awful
besides
the
fact
that
we're
copying
and
pasting
things
over
and
over
again
is
time.
B
You've
got
a
lot
of
auditability
and
compliance
requirements.
How
do
you
actually
know
that
all
of
those
teams
are
following
the
same
software
delivery
process,
a
little
less
academic
way
to
say
that
is
like
I
said
you
all
have
to
go,
do
container
image
scanning?
How
do
I
actually
know
that
you're
doing
that
and
then
sustainment?
No
one
gets
the
pipeline
right
on
the
first
try.
A
B
It
is,
and
let's
let's
talk
about
a
better
way
right.
So,
despite
the
fact
that
all
these
different
teams
are
using
different
tools,
what
we
drew
on
the
whiteboard
didn't
didn't
care
what
type
of
application
it
was.
We
were
able
to
say
in
tool
agnostic
terms,
what
does
what's
the
business
process
to
get
code
from
a
developer's
laptop
to
production
as
quickly
as
possible
without
sacrificing
code
quality
or
security?
B
A
B
That's
exactly
right,
so
here
we've
got
jenkins
using
the
upstream
jenkins
helm,
chart
nothing
fancy.
Here
we
installed
the
jenkins
sampling
engine.
We
configured
it
to
point
to
a
template,
so
let's
actually
kick
off
a
build
here
refresh
the
page
and
all
this
is
running.
Let's
go!
Take
a
look
at
what
a
pipeline
template
looks
like
in
the
janket
sampling
engine
and
how
this
actually
gets
implemented
so
I'll
zoom
in
a
bit.
This
is
probably
pretty
small
right
now.
B
There
we
go
so
if
you're
like
me,
you've
written
a
lot
of
jenkins
files
in
your
time
and
they
usually
balloon
out
into
700
line
files
where
you're
trying
to
represent
the
business
logic
of
your
pipeline
right.
So
when
we,
when
we
draw
pipelines
on
the
whiteboard
they're
linear,
you
say,
build
test
scan,
deploy
in
reality,
they
are
not
linear.
They
map
to
the
branching
strategy
or
the
way
developers
are
collaborating
on
a
code
base
right.
B
So
if
you
want
to
change
what
types
of
tests
happen
on
a
pull
request
to
a
development
branch
or
you
want
to
change
which
application
environment
you're
deploying
to
when
you
merge
a
pull
request,
all
of
that
logic
typically
ends
up
in
the
same
place,
where
you're
defining
exactly
how
you're
going
to
perform
the
different
types
of
tests
we
talked
about.
So
that
turns
into
a
700
line
file
the
jenkins
sampling
engine
sort
of
turns
out
on
its
head.
The
key
value
here
is
that
we've
been
able
to
separate
the
business
logic
of
your
pipeline.
B
So
what
should
happen?
And
when
from
the
technical
implementation
of
exactly
what
tool
do
I
care
about?
Or
is
going
to
implement
these
steps
for
a
particular
microservice
right?
So
if
we
sort
of
walk
through
this
7
17
line
pipeline
template
in
parallel
we're
going
to
run
unit
testing
and
static
code
analysis,
then
we're
going
to
run
application
dependency
scanning,
then
we're
going
to
do
a
build
and
scan
a
container
image,
we're
going
to
deploy
it
to
a
production
environment
and
then,
in
parallel,
we're
going
to
do
penetration,
testing
and
accessibility
compliance
scanning.
B
So
one
of
the
most
important
things
to
realize
about
this
template
is
those
step
names.
Unit
tests
and
static
code
analysis
are
generic
on
purpose.
It
doesn't
matter
if
unit
test
comes
from
a
maven
library
or
comes
from
a
gradle
library
or
an
npm
library,
application
dependency
scan
can
come
from
os
dependency
checker,
it
can
come
from
nexus,
firewall
build,
can
come
from
docker
or
image
or
any
of
the
17
tools
out
there
to
build
a
container
image.
These
days,
scanning
container
image
can
come
from
twist,
lock
or
systig
or
anchor
or
claire.
B
So
the
point
is
that
this
single
tool,
agnostic
pipeline
template,
can
be
reused
for
every
single
microservice
or
even
every
single
application
in
your
organization,
but
still
be
flexible
enough
to
choose
the
right
tools
for
the
job
based
upon
what
type
of
application
you
are.
So,
alongside
these
pipeline
templates,
you
can
define
hierarchical
pipeline
configuration
files.
B
B
B
The
whole
point
is
that
you're
able
to
plug
and
play
with
these
different
library
implementations,
these
libraries
can
take
configuration
options
right.
So,
as
a
devops
engineer,
there's
no
reason
that
every
single
devops
engineer
that
needs
to
write
a
pipeline
should
have
to
google
sonarq
plus
jenkins
and
find
the
15
lines
of
code.
It
takes
to
implement
static
code
analysis
from
a
jenkins
pipeline
right
so
by
modularizing
our
pipelines
into
these
pipeline
libraries
and
externalizing
configuration
through
these
library
parameters.
B
B
So
if
we
head
back
to
jenkins
we'll
see
if
our
our
pipeline
is
done
yet
so
it's
still
running,
but
we've
got
some
examples
that
are
done.
So,
let's
talk
about
each
of
these
steps
really
quickly
and
why
they're
important
unit
testing.
At
this
point,
I
hope
that
one's
pretty
clear
we
need
to
make
sure
that
our
code
actually
works
at
the
unit
level
from
a
security
standpoint
application
dependency
scanning.
B
We
need
to
make
sure
that
the
materials
we're
bringing
into
our
application
all
those
third-party
dependencies
don't
have
known
vulnerabilities
right.
So
there
was
a.
There
have
been
some
pretty
significant
data
breaches
in
the
last
couple
years.
I
won't
name
anyone
by
name
but
a
lot
of
times.
If
you
go
look
at
what
exactly
caused
this,
it
would
be
a
insecure
application
dependency
and
there
was
no
application
dependency
scanning
being
done
right.
B
So
the
next
step
there
is
static
code
analysis,
so
application
dependency
scanning.
If
we
go
back
and
look
at
the
slides
that
we
had
to
talk
about
the
big
picture,
tasks
that
we're
doing
static
code
analysis
is
cool.
My
raw
materials,
my
dependencies
are
secure.
Let's
make
sure
my
code
itself
is
secure
and
sonarcube
is
a
great
tool
to
be
able
to
do
that
with
containerization
comes
a
new
artifact
in
the
container
image.
B
Pulling
an
image
from
docker
hub
is
the
same
thing
as
going
to
finding
a
sketchy
van
in
like
a
walmart
parking
lot
and
putting
that
in
your
production
environment.
So
you
need
to
make
sure
that
those
environments
are
secure
and
that
they're
as
small
as
possible
and
only
have
what
you
need
to
run
your
application.
B
Continuous
compliance
right,
so
everything
up
until
now
has
been
the
app
layer,
but
we're
running
this
app
on
infrastructure.
We
need
to
make
sure
that
those
underlying
servers
are
compliant
and
there's
a
ton
of
you
know,
profiles
out
there
that
that
have
best
practices
baked
into
them
for
how
to
configure
your
infrastructure.
B
A
lot
of
the
tools
that
do
container
image
scanning
can
also
do
continuous
compliance
to
make
sure
those
hosts
are
secured,
dynamic
application
security
testing.
It's
a
really
long
way
to
say
pen
testing
right,
so
that's
actually
attacking
a
deployed
application
and
seeing
if
it's
susceptible
to
common
exploitations
and
then
accessibility
assurance.
So
we
need
to
make
sure
that
the
applications
we're
building
are
accessible
to
everybody,
obvious
examples
of
that
are
we
included
in
an
image
but
never
included
in
alt
text,
so
there's
no
tool
out
there.
B
So
if
we
go
back
to
jenkins
each
of
the
steps
of
the
pipeline
that
were
generating
results
were
archiving
those
artifacts
right,
so
we'll
get
reports
from
owasp
and
we'll
get
reports
from
google
lighthouse
and
owasp
zap,
which
are
doing
pen,
testing
and
accessibility
compliance
testing,
and
this
gives
us
an
audit
trail
right.
So
every
time
we
make
a
change,
we
can
then
go
see
if
what
the
sorry,
what
the
resulting
scans
were
for
that.
B
So
at
the
end
of
the
day,
new
cves
come
out
every
day.
Even
if
you
did
all
of
these
great
kinds
of
security
testing,
you
still
need
to
make
sure
that
you're
monitoring
your
production,
environment
and
there's
some
really
great
tools
that
can
help
with
that,
and
I
pop
is
going
to
to
walk
through
how
we
can
make
sure
our
clusters
are
secure
at
runtime.
A
And
so
you
know,
the
term,
I
would
say,
is
more
runtime
security
than
monitoring.
Obviously
we
have
the
modern
capability
with
with
systig,
but
if
we're
talking,
you
know
full
runtime
capability.
The
idea
that
we
would
do
here
is
steve
if
you
could
stop
the
share.
So
steven
mentioned
runtime
security
being
important
and
that's
where
falco
comes
into
play,
and
so
falco
is
an
incubated
cncf
project,
hey
we're
at
kubecon
right,
it's
a
cncf,
you
know
event
right
and
so
the
way
that
falco
works
is
there's
your
app
right,
which
sits
on
kubernetes.
A
In
this
case
we
have
a
g
jk
cluster,
and
then
you
have
your
runtime
and
so
what's
happening.
Is
we
have
this
ebpf
function
or
linux
driver
that's
tapping
into
like
capsis
p
trace
to
be
able
to
then
evaluate
at
a
runtime
assertion.
You
have
a
set
of
security
rules
and
these
can
be
said
as
yaml.
We
give
you
a
bunch
of
them
out
of
the
box
if
you'd
like
to
contribute
more
security
rules,
they're
pretty
easy.
A
It's
a
you
know,
easy
syntax
for
you
to
do
so,
and
then
you
can
then
take
this
and
send
these
alerts
out
now
right
now,
we're
probably
going
to
do
it
to
standard
out,
but
you
can
do
it
out.
The
grpc
client
go
client
rust,
prometheus.
Another
project
that
we've
integrated
I'll
show
an
example
of
is
a
falco
sidekick
shout
out
to
the
team
for
that,
and
so
you
have
all
of
these
pieces
that
you
can
kind
of
tie
these
things
together
again
get
your
runtime
assertation.
A
B
I'm
gonna
be
honest
right
when
I
heard
about
falco.
I
was
like
that's
awesome,
but
I
don't
know
what
capsis
p
trace
is,
but
the
way
falco
has
created
these
abstractions
with
yaml
files
to
create
rules
makes
it
accessible
to
anybody
like.
Even
I
can
write
these
ammo
rules
for
falco
and
securityhub
makes
it
even
better.
So
I
don't
have
to
write
these
rules
from
scratch.
I
can
go
to
you,
know
community
source
rules
to
get
best
practices
right
out
of
the
box.
A
Stephen,
it's
like
we've
done
this
before
it's
amazing.
What's
your
name
again,
all
right,
so
I'm
gonna
share
my
screen
again
just
let
me
clear
my
terminals
yellow
all
right,
let's
go
ahead
and
do
some
attacking
here
all
right.
So
here's
what
I'm
doing!
I'm
I've
I've
tailed
my
the
the
same
server
that
is
sitting
where
I've
attached
this
this
xiaomi
file
that
or
this
html
file
that
I'm
going
to
copy
into
the
directory
right.
So
let's
go
ahead
and
do
that
and
we're
seeing
there's
an
attachment
to
it.
A
A
So
with
that
being
said,
we
just
got
attacked
and
we
also
might
have
got.
We
used
the
power
of
slack
to
be
able
to
er
the
power
of
the
sidekick
to
be
able
to
send
that
output
out
to
here-
and
I'm
saying,
oh,
my
god,
stephen
we
got
hacked
now
we
got
to
figure
out
what
happened
so
again.
We've
used
the
falco
tool
to
do
that:
runtime
detection
and
then
we
have
the
the
capability
to
send
those
those
rule.
The
you
know
the
rule
ascension
that
the
rule
sets
that
are
there.
A
Then
we
have
the
ability
to
send
the
standard
out
or
out
the
grpc
or
out
to
the
the
other
things
that
I
showed
as
part
of
the
the
slide
there.
So
that's
pretty
much
again
in
a
nutshell,
all
the
things
you
all,
the
goodness
that
you
can
do
for
a
runtime
security
perspective.
Let
me
pass
it
over
to
my
friend
mr
steven.
B
All
right,
so
you
guys
are
really
getting
your
bang
for
your
buck
this
session.
I
got
one
more
demo
for
you,
so
we
talked
about
application,
security
and
building
a
tool
agnostic,
devsecops
pipeline
that
you
can
share
across
teams.
We
talked
about
you
know
it's
that's
still
not
good
enough.
You
need
to
do
runtime
security
with
falco,
and
even
that
is
not
good
enough
right.
All
the
best
security
in
the
world
is
not
going
to
help
you
if
you
still
have
insecure
configurations.
B
So
if
we
take
a
look
at
some
examples
here
with
gatekeeper,
you
create
what
are
called
constraint,
templates
and
constraints.
So
if
I
take
a
look
at
this,
this
constraint
template
that
we've
got
here.
I'm
able
to
create
generic
regal
policies
called
constraint,
templates
that
define
a
new
custom
resource
definition.
In
this
case,
it's
called
kate's
required
labels
and
down
there
at
the
bottom,
in
the
in
the
keys
and
values
for
the
rego.
We
have
a
policy
that
says
every
single
object
that
conforms
to
this
constraint.
B
Template
must
have
a
corresponding
label,
but
through
the
constraint
template
it's
generic
right.
So
we're
going
to
be
able
to
create
multiple
constraints
from
this
single
constraint,
template
to
lock
down
our
cluster
even
further.
So
one
particular
example
of
that
is
if
we
wanted
to
say
that
every
single
namespace
must
have
a
gatekeeper
label
on
it.
B
So
right
now
I
have
a
namespace
file,
no
labels
on
it
right,
so
this
would
not
be
allowed
through
opa
and
through
gatekeeper.
So
if
I
say
cube
apply,
hyphen,
f,
namespace,
I'm
gonna
get
an
error
message
that
tells
me
this
request
has
been
denied.
You
must
provide
the
labels
for
gatekeeper
all
right,
so
I
can
go
edit.
This
namespace
file
make
it
conformant
to
the
policy
that
we
just
created,
so
I
can
create
labels.
I
can
make
sure
that
I
have
the
gatekeeper
label
that
is
required.
B
B
So
when
does
this
come
into
play,
common
use,
cases
for
opa
would
be
things
like.
Every
deployment
must
have
resource
requests
and
limits.
I
get
a
lot
of
calls
of
hey
my
cluster's
broken
come
help
me
fix
it
and
most
of
the
time
it's
because
people
never
put
resource
requests
or
constraints
on
their
containers,
so
the
scheduler
blows
up.
You'll
have
things
like
privileged
containers
and
pod
security
policies
right.
B
So
there's
all
these
best
practices
that
we've
learned
along
the
way,
as
we've
bricked
our
clusters
that
we
want
to
be
able
to
put
guard
rails
on
future
clusters
to
help
people
deploy,
resources
that
are
going
to
be
reliable.
An
open
policy
agent
gives
us
a
way
to
codify
those
best
practices
and
security
policies
and
then
enforce
them
throughout
the
cluster
at
a
super
granular
level.
So,
in
this
talk
today
we
learned
about
all
the
different
kinds
of
security
testing
steven.
A
A
I
want
to
show
you
this
so
notice
when
stephen
I'm
going
to
show
you
this
notice,
when
stephen
did
his
his
creation
of
this
testing
namespaces
another
beautiful
thing.
Remember,
I
told
you
about
the
kubernetes
audit.
We
were
able
to
basically
look
at
this
and
see
that
you're
not
allowed
to
be
listeners
to
be
able
to
be
creating
name
spaces.
So
this
is
an
example
where
we're
using
opa
and
falco
together
to
be
able
to
understand.
A
B
So
we
we
covered
a
lot
today
in
a
half
hour.
Everything
that
we
talked
about
was
how
to
integrate
security
into
every
step
of
your
software
development
life
cycle
and
build
a
devsecops
pipeline
through
things
like
application,
dependency,
scanning
static
code,
analysis,
container
image,
scanning,
continuous
compliance,
penetration,
testing,
accessibility,
compliance
testing,
then
we
we
talked
about
configuration
governance
right
so
being
able
to
have
really
granular
control
over
the
types
of
kubernetes
objects
that
are
being
created,
and
then
we
talked
about
runtime
security.
So
pop
you
want
to
debrief
here
on
on
runtime.
A
A
We,
you
know
when
we
terminaled
into
this
into
the
container
when
we
had
the
you
know
the
example
of
opa,
where
we
were
able
to
like
look
at
that
and
see
that
steven
went
in
there
through
using
cube
audit
rules
that
we
had
attached
to
it
and
then
sending
that
output
either
to
stand
it
out
on
your
screen
or
to
grpc
or
to
sla
slack
or
any
of
the
things
using
the
falco
sidekick
project.
So
all
of
that
again,
we've
done
that
through
the
power
of
open
source
and.