►
From YouTube: Building Reusable DevSecOps Pipelines on a Secure Kubernetes... Steven Terrana & Michael Ducy
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Building Reusable DevSecOps Pipelines on a Secure Kubernetes Platform - Steven Terrana, Booz Allen Hamilton & Michael Ducy, Sysdig
Onboarding development teams can often be the critical point in determining if a team will adopt modern Cloud Native and DevSecOps practices. If there is too much friction for developers to build, scan, and test their applications or to secure their application environments then these best practices are often pushed aside. In this talk we’ll cover how we automated the creation of a trusted software supply chain. Through a live demonstration, we will show how this approach accelerates adoption by allowing developers to inherit a preconfigured pipeline performing various security tests (and underlying tooling) as well as safeguards (via the CNCF Sandbox project Falco) put in place to monitor production workloads for security problems.
https://sched.co/UaWr
A
Thank
you
for
for
coming
to
our
talk
today
we're
going
to
talk
about
how
to
secure
applications
through
defense-in-depth
by
building
a
trusted
software
supply
chain
and
then
having
continuous
monitoring
on
your
grantees
clusters.
So
a
quick
introduction,
I'm
Stephen,
Tirana
I'm,
a
lead
technologist
at
Booz,
Allen,
Hamilton
or
a
federal
consultancy.
So
I
do
a
lot
of
legacy.
It
modernization
within
the
government
space
and.
B
A
All
right
so
vulnerabilities,
this
is
a
representation
of
the
typical
vulnerability
lifecycle
right.
So,
if
you
think
about
the
average
vulnerability,
there's
going
to
be
a
time
when
that
vulnerability
is
introduced
into
your
environment,
there's
going
to
be
a
time
when
an
exploitation
for
that
environment
is
for
that.
Vulnerability
is
discovered
at
some
point.
You're
gonna
realize
that
your
environment
is
compromised
and
then,
presumably
after
that
you're
going
to
patch
that
vulnerability
right
so
that
time
between
when
that
exploitation
is
discovered
and
when
you
actually
patch
it
is
when
you're
most
susceptible
to
being
compromised.
A
So
what
can
we
actually
do
about
this
through
dedsec
ops
and
integrating
security
into
your
software,
home
and
life
cycle?
You
can
shrink
that
time
from
when
an
exploitation
is
discovered
to
when
you're
able
to
identify
that
it's
in
your
environment
and
then
subsequently
patch
it
alright.
So
everyone
I've
ever
talked
to
has
a
different
definition
of
DevOps
or
devstack
ops.
So
this
slide
search
the
level
set.
A
Traditionally,
DevOps
was
about
getting
application
developers
and
operations
engineers
to
work
together
more
effectively
through
automated
testing
infrastructure
as
code,
so
we're
able
to
quickly
and
reliably
deploy
our
code,
we're
fairly
confident
that
it's
going
to
work
through
automated
testing,
but
we
were
still
experiencing
bottlenecks
when
it
came
time
to
act.
We
deploy
to
production
and
that's
because
we
never
actually
pulled
in
the
security
team.
A
So
dead-set
cops
is
all
about
incorporating
the
security
team's
requirements,
there's
a
whole
bunch
of
different
kinds
of
security
testing
that
you
can
incorporate
like
application
dependency
scanning,
making
sure
third-party
libraries
are
secure,
static
code
analysis,
making
sure
that
you
know
you
haven't
hard-coded
IP,
addresses
or
passwords
into
your
code
base
dynamic
application.
Security
testing
is
a
mouthful
of
a
way
to
say
penetration
testing
so
actually
attacking
your
deployed
applications
and
seeing
if
they're
susceptible
to
comment.
Exploitations
with
containerization
comes
the
container
image.
A
So
there's
container
image
gaming
to
make
sure
that,
if
we're
pulling
images
off
docker
hub
with
no
clue
who
actually
built
them,
that
they
don't
have
known
vulnerabilities
or
Bitcoin
miners,
Oh
bonor
ability
is
in
general,
come
in
two
flavors
you've
got
vulnerable
packages
and
vulnerable
configurations.
So
a
lot
of
the
container
image
scanning
tools
out
there
will
help
you
out
with
the
compliance
side,
which
is
making
sure
that
you
haven't
configured
your
containers
to
run
as
route.
A
You
know
making
sure
that
you're
compliant
with
different
federally
regulated
guidelines
like
NIST
or
secure
technical
implementation
guides.
If
you
happen
to
be
working
in
the
federal
space
like
I,
primarily
do
and
in
that
same
vein
of
compliance,
you've
got
accessibility
assurance
so
making
sure
that
the
applications
we're
building
are
accessible
and
508
compliant.
It
is
important
to
note
there's
no
tool
out
there.
That's
going
to
tell
you
that
you
are
100%
508
compliant,
but
what
they
can
do
is
tell
you
if
you're,
not
all
right.
A
So
if
you
included
an
image
on
your
website,
for
example,
that
doesn't
have
an
alt
text,
that's
a
pretty
black
and
white,
easy
to
figure
out
508
compliance
issue.
So
a
lot
of
the
tools
that
can
help
you
with
this
are
primarily
trying
to
give
developers
fast
feedback
on
accessibility
issues
on
their
application,
so
that
those
that
are
doing
the
manual
testing
can
focus
on
the
more
complex
aspects
of
compliance.
A
So
all
of
the
security
testing
is,
in
addition
to
all
of
the
other
kinds
of
quality
assurance
testing
that
the
industry's
been
doing
for
for
a
couple
years
now.
So
things
like
unit
testing,
functional
testing,
surveys,
test
automation,
pull
a
word
out
of
the
dictionary
and
add
testing
to
the
end
of
it.
A
If
you're
supporting
enterprise
scale
pipeline
development,
so
you're
a
centralized
DevOps
team
helping
support,
CI,
CV
pipelines
for
multiple
applications
at
the
same
time
or
if
you're,
building
a
platform
to
host
micro
services,
it's
likely
that
applications
in
your
portfolio
or
leveraging
different
tools.
Alright,
so
your
front-end
applications
might
be
doing.
You
know
testing
with
something
like
karma
or
jest,
whereas
back-end
applications,
if
they're
Java,
might
be
using
j-unit,
you
could
have
a
Python
back-end.
Some
teams
in
your
organization
might
be
using
sonar
cube
for
static
code
analysis.
A
Other
teams
might
be
using
fortify,
so
the
current
CIS
CD
landscape
out
there
is
primarily
targeted
at
application,
specific
pipelines.
Most
of
the
time
you
get
a
artifact
in
your
source
code
repository
that
outlines
as
code
what
your
software
delivery
processes
look
like.
So
when
a
developer
does
something
in
github
or
whatever
your
source
code
provider
is
what
does
that
mean
from
a
pipeline
perspective?
So
when
you're
supporting,
for
example,
a
60
micro
service
based
application?
That
means
that
you
have
60
different
pipelines
that
exist
right.
A
Wrung
from
stack
overflow,
we
have
all
googled
Stone
our
cube,
plus
Jenkins,
and
found
the
six
lines
of
code
to
do
a
static
code
analysis.
If
you
want
to
make
a
change
to
that,
because
we're
we're
all
striving
to
be
continuously
improving
our
software
delivery
processes.
That
means
you
have
to
go
open,
64,
requests
right,
and
you
need
to
work
with
these
application
development
teams
to
coordinate
a
migration
to
a
new
version
of
the
pipeline
and
having
to
have
done
this
myself,
it
can
be
an
extremely
cumbersome
process
that
ultimately
introduces
so
much
technical
debt.
A
They,
you
kind
of
just
leave
the
pipeline,
as
is
you,
don't
want
to
add
new
tools,
and
it
really
starts
to
stagnate
your
ability
to
continuously
improve
and
at
Booz
Allen.
We
do
a
lot
of
application
modernization.
Every
project
that
we're
working
on
needs
a
pipeline.
So
this
duplication
of
effort
across
applications
takes
place
within
a
single
project.
It
happens
your
ability
to
continuously
improve,
but
then
we're
also
reinventing
the
wheel
across
every
project
that
needs
a
pipeline.
A
So
every
DevOps
engineer,
Booz
Allen,
would
go
to
Google
or
Stack
Overflow
and
find
the
Sam
code
snippet
and
introduce
it
so
about
three
years
ago
we
set
out
to
figure
out
like
how.
How
can
we
make
this
go
faster?
You
know
when
a
project
starts
up.
Not
everyone
is
a
Jenkins
expert.
I
know
way
more
about
Jenkins
and
I'd
like
to
admit
and
really
should
have.
It
would
take
four
to
six
months
to
find
all
the
right,
Stack
Overflow
snippets,
to
tie
together.
A
So
we
wanted
to
drastically
improve
how
quickly
we
could
deploy
pipelines
while
also
being
able
to
introduce
a
level
of
organizational
governance
right.
One
of
the
biggest
issues
with
those
60
different
Jenkins
files
is
that
you
don't
actually
know
they're
all
doing
the
same
thing
right
if
in
order
or
is
it
if
an
organization
says
you
need
to
do
container
image
scanning,
you
need
to
do
penetration
testing
you're,
really
just
trusting
that
all
of
the
different
development
teams
have
implemented
a
pipeline
that
meets
these
organizational
standards.
I
mean
we
wanted
to
figure
out.
A
How
could
we
take
pipeline
development
from
what
you
see
on
the
screen,
which
is
every
development
team,
has
a
pipeline
that
largely
follows
the
same
steps.
You
know
it
doesn't
matter
if
you
are
doesn't
matter
which
tools
you're
using
the
process
is
largely
the
same.
You're
gonna
build
something
you're
gonna
package,
it
somehow
you're
gonna
test
it
deploy
it
scan
it.
A
So
that's
really
really
the
revelation
that
we
came
to
it
was
what,
if
we
could
create
pipelines
in
a
tool,
agnostic,
templated
way
to
take
pipeline
development
from
what
you
see
on
the
screen,
so
something
that
looks
a
little
bit
more
like
this.
Where
you
have
a
centralized
DevOps
team,
that's
supporting
multiple
applications.
They
maintain
a
tool
agnostic,
templated
workflow,
that
outlines
the
business
approved
software
delivery
processes.
A
You
know
this
makes
it
really
convenient
when
working
with
security
teams,
making
sure
that
the
requirements
are
represented
because
they
can
be
stakeholders
and
coming
up
with
this
pipeline
template.
That
team
would
likely
consist
of
all
of
the
different
stakeholders
that
go
into
your
software
delivery
processes.
So
that
includes
like
CEQA
engineers,
DevOps
engineers
that
know
how
to
write
pipelines.
Qa
engineers
that
can
speak
to
test
automation.
So
what
if
we
had
a
centralized
team
that
could
define
this
tool?
Agnostic
template?
A
So
let's
say
you
wanted
to
build,
scan,
deploy
and
test
the
different
tools
that
are
gonna
perform.
These
actions
of
building
scanning
deploying
testing
can
just
be
modularized
into
a
different
pipeline
library.
We
call
it
so
if
I
wanted
to
do
a
scan
with
static
code
analysis,
tatak
code
analysis
as
a
scan
with
sonar
cube,
there'd
be
a
Sun
or
tube
library
that
introduces
a
static
code
analysis
step
to
my
pipeline
and
there'd
be
a
fortify
library
that
introduces
the
static
code
analysis
step
to
my
pipeline.
A
So,
alongside
your
pipeline
template
you
get
a
pipeline
configuration
which
just
tells
jenkins
or
the
pipeline.
What
tools
am
I
using
to
actually
build
this
dedsec
ops
pipeline?
There's
some
organizational
governance
there.
So
if,
at
the
top
level,
you
have
some
tools
that
you
want
to
enforce,
everybody
is
using.
A
You
can
consolidate
those
common
controls
to
a
centralized
repository,
but
still
give
development
teams
the
flexibility
they
need
to
customize
the
pipeline
to
introduce
different
tools
right
so
that
that
centralized
pipeline
definition
of
build
scan
deploying
tests
can
turn
into
what
you
see
at
the
bottom
of
the
screen,
which
is
building
with
maven
scanning,
with
fortify
deploying
with
ansible
and
testing
with
selenium,
or
it
can
mean
building
a
docker
image
or
container
image
scanning.
It
was
shown
r-cube
deploying
it
with
helm
and
testing
an
api
with
something
like
rest
assured.
A
A
The
FOB
finger
I
will
do
that.
Thank
you
Michael.
Can
everyone
see
okay,
seeing
general
nods
of
agreement?
Yes,
so
this
is
a
pipeline
job.
In
Jenkins
we
wrote
a
Jenkins
plugin
called
the
Jenkins
templating
engine
which
achieves
a
lot
of
what
I'm
talking
about
the
ability
to
take
that
Jenkins
file
that
application-specific
out
of
individualized
source
code
repositories,
define
it
in
a
central
place
and
allow
teams
to
pull
in
their
unique
configurations.
So
what
you
see
on
the
screen
is
an
example
of
a
simple
pipeline
template.
A
They
can
get
a
lot
more
complex
if
you
want
to
map
your
your
pipeline
to
your
branching
strategy.
So
when
you
do
merges
to
the
develop
branch,
it
does
something
different
than
merges
to
the
master
branch,
but
for
the
sake
of
demonstration,
let's
assume
you've
got
a
linear
pipeline.
That's
going
to
do
build
some
static
code
analysis
and
it's
going
to
deploy
to
a
dev
and
prod
environment.
Alongside
that
pipeline
template
you
get
a
pipeline
configuration
file,
so
this
is
going
to
implement
that
pipeline
template.
A
If
you
will
so
there's
a
library
section,
that's
going
to
pull
in
the
specific
tools
that
you're
using
to
implement
the
pipeline.
So
in
our
case
these
libraries
are
going
to
do
print
statements
just
because
we
want
to
really
demonstrate-
and
you
know,
communicate
what
pipeline
template
looks
like
so
they're
gonna
do
some
print
statements
to
say
which
library
contributed,
which
step
alongside
your
libraries,
you
can
dynamically
define
application
environments
every
one
every
project
I've
ever
worked
on
has
a
different
number
of
application
environments.
They
give
them
unique
names.
A
I've
been
on
a
project
where
dev
was
called
squirtle
I.
Don't
know
why!
But
from
your
configuration
file
you
can
dynamically
define
these
app
environments
and
store
whatever
environmental
context
you
need
to
know
in
order
to
perform
deployments
to
those
specific
environments.
So,
if
I
apply
this
configuration
zoom
out
for
a
second
and
click,
build
I'll,
look
at
a
finished
one,
so
we
don't
the
wait
for
it
to
load.
A
But
what
we're
going
to
notice
is
that
at
the
beginning
of
the
pipeline,
jte
Jenkinson
playing
engine
for
short
is
going
to
initialize
the
pipeline,
so
we're
gonna
go
pull
in
all
the
different
libraries
that
implement
these
steps.
These
libraries
are
just
modularized
tool,
integrations
so
they're
stored
in
a
source
code.
Repository
they're
just
directories
in
your
repo.
So
for
example,
we
loaded
the
maven
library
there's
a
build
act,
ruby
step
that
turns
into
the
build
step
when
you
load
it
in
your
in
your
pipeline.
A
So
if
we
take
a
look
at
this
build
step,
it's
just
Jenkins
pipeline
is
code,
so
the
framework
introduced
is
some
syntactic
sugar
to
make
it
easier
to
write,
libraries
and
access
configuration
from
your
config
file,
but
jte
is
really
just
a
different
way
of
organizing
your
pipeline
code
to
achieve
this
pipeline,
templating
and
reuse.
So
if
we
go
back
over
to
Jenkins,
we
loaded
all
the
requisite
libraries
that
we
talked
about
the
stage
view
will
probably
give
us
a
easier
to
parse
representation
where
we
say
that
we
did
a
build
with
maven.
A
We
did
sonar,
cube,
static
code
analysis
and
then
we
deployed
to
a
dev
in
production
environment,
so
in
practice,
you're
not
gonna
want
to
put
that
information
in
your
Jenkins
UI
you're
gonna
store
it
in
your
repository
most
likely.
So
this
is
a
multi
branch
project,
which
means
it's
going
to
automatically
create
jobs
for
every
branch
and
pull
request
in
your
pipeline.
A
Here,
the
Jenkins
file
has
been
pulled
out
of
the
repository.
We
can
store
it
right
alongside
our
libraries
or
in
a
separate
repo,
doesn't
doesn't
matter.
We
can
define
that
pipeline
template
externally,
alongside
a
pipeline
configuration
that
just
says
this
is
what's
required
by
the
organization
we
have
this
nice
key
that
says,
merge
equals
true,
which
is
going
to
allow
the
application
to
just
add
in
the
maven
library
and
when
we
build
it.
Every
branch
of
this
pull
request
of
this
repository
excuse
me
is
going
to
do
the
exact
same
thing.
A
So
now
the
real
power
comes
when
we
want
to
scale
this
up
to
multiple
applications.
So
the
idea
is
the
same.
What
if
I
had
a
Gradle
application
instead
of
a
maven
application?
All
I
would
want
to
change.
Is
the
implementation
of
my
build
step
for
this
example,
so
the
the
maven
application
would
have
a
pipeline
configuration
file
that
that
it's
pulling
in
the
Maven
library?
We
can
take
a
look
at
this
repo.
All
it
says,
is
hey
I'm
using
maven.
A
We
can
go,
take
a
look
at
the
Gradle
repository
and
they
just
say
that
they're
using
Gradle,
but
if
we
go
look
at
Jenkins
when
you
look
at
the
Gradle
application,
the
pipeline
pulled
in
the
Gradle
library,
so
we
had
a
Gradle
implementation
of
our
build
step.
We
can
go
over
and
take
a
look
at
the
maven
library
and
see
that
it's
the
same
for
its
using
maven
right.
So
what's
the
big
picture
here,
if
we
go
back
to
our
slides.
A
So
what
are
the
main
benefits
of
this
organizational
governance?
You
don't
have
to
copy
and
paste
Jenkins
files
anymore.
You
don't
all
have
to
figure
out
how
to
do
static
code
analysis
with
Soner
cube
by
googling
it
and
copying
and
pasting
it
so
organizational
governance
centralize
your
pipeline
definition
to
a
common
place
and
define
processes
that
are
approved
by
your
security
teams,
regardless
of
specific
tool
implementations
across
your
entire
organization.
Optimizing
pipeline
code
reuse,
so
these
libraries
are
reusable
and
open-source
at
Booz
Allen
we've
seen
pipeline
development
decreased
by
97%
for
new
projects.
A
If
you're
using
existing
tool
integrations
because
the
works
already
been
done,
I
we
can
reuse
these
libraries
across
all
the
pipelines
that
we're
developing
to
drastically
speed
up
pipeline
development
from
something
like
four
to
six
months
to
four
to
six
days,
if
you're
following
a
happy
path
and
then
simplifying
pipeline
maintainability.
In
my
humble
opinion,
it's
a
lot
easier
to
manage
a
centralized
pipeline
template
with
modular
eyes
tool
integrations
than
it
is
to
keep
straight
copied
and
pasted
sixty
different
versions
of
a
pipeline
as
code
artifact
and
handle
migrations
of
improving
that
over
time.
A
B
You
Steven,
so
if
we
think
about
this
pipeline,
the
one
thing
that
you
want
to
do
when
you
start
looking
at
your
pipelines
in
this
way
is
that
starting
to
remove
areas
of
waste
and
areas
of
waste
tend
to
be
areas
where
you
spend
a
lot
of
time
right.
So
the
pipeline
definition
that
he
showed
here
is
actually
really
really
useful
exercise
to
do
to
actually
draw
out
and
put
times
on
each
one
of
those
areas,
because
then
it
starts
helping.
B
You
understand
where
you
need
to
optimize,
but
also
when
you
look
at
the
entire
lifecycle
like
this,
we
were
like.
How
can
we
reduce
this
time
to
where
we
actually
detect
the
exploitation
or
discover
the
exploitation
exploitation?
A
vulnerability
has
been
discovered
and
then
we've
been
hacked,
or
we
have
to
go
through
this
entire
cycle
again
to
actually
go
and
actually
patch
our
applications
and
the
challenges
is
with
containers
and
ephemeral
workloads.
B
So
we
did
a
study
at
Cystic
recently
and
containers
are
lasting
on
average
about
five
minutes,
and
so
in
a
container
that
only
lasts
five
minutes.
How
can
you
actually
discover
that
it
was
breached?
Someone
was
able
to
able
to
do
data
exfiltration
and
other
things
like
that,
as
well.
So
with
that
we'll
introduce
Falco,
and
so
what
is
Falco?
So
it's
a
behavioral,
Activity
Monitor.
It
detects
suspicious
activity
defined
by
a
set
of
rules
in
non-marketing
speak.
B
You
can
call
it
a
host
intrusion
detection
system,
but
what
makes
us
unique
is
that
were
focused
on
container
and
Orchestrator
support,
so
we
have
native
integrations
into
docker
cryo
container
d
kubernetes.
It's
actually
pull
back.
The
metadata
information
from
the
from
those
sources
combine
that
with
what
we're
actually
seeing
on
a
system
call
level
and
in
the
kubernetes
audit
event
log
level
and
using
that
to
actually
create
very,
very
rich
and
granular
rules,
and
when
we
detect
something
that's
abnormal
will
alert
via
various
methods
and
it's
open
source.
B
So
we're
a
CNC
F
sandbox
project.
You
can
contribute
improvements
to
the
project
or
rules
as
well.
Speaking
of
that,
we're
also
going
up
for
incubation
as
well,
and
we
have
incue
bation
proposal
up
so
hopefully
within
the
next
month
or
so
once
everyone
gets
past.
Coop,
con
and
thanksgiving
will
be
moving
into
incubation,
which
means
the
CNC
F
will
be
supporting
us
even
more
with
infrastructure
and
other
things
like
that
as
well.
B
So,
since
containers
are
supposed
to
be
one
process
for
container
and
it's
an
isolated
process,
the
processes
as
I
say
are
scoped
as
to
what's
expected.
So
you
know
what
processes
are
going
to
run.
You
know
what
files
they're
going
to
manipulate.
You
know
what
directories
are
going
to
touch.
You
know
what
ports
they're,
gonna,
listen
on
and
inside
of
the
container,
it
should
be
very,
very
easy
to
define
all
of
the
activity
from
a
system
call
level
that's
going
to
take
place
when
you
deploy
it.
B
The
challenges
is
that
containers
are
immutable,
so
the
container
image
itself,
if
you
change
something,
if
you
change
the
configuration
and
rebuild
the
container,
the
SHA
changes.
So
it's
a
new
container,
however,
when
the
container
launches
that
that
environment
can
still
be
manipulated,
very
few
people
run
containers
in
a
read-only
root,
filesystem
mode,
and
so,
if
somebody
actually
breaches
the
container,
they
can
start
to
actually
add
in
things.
So
they
can
do
an
app
get
upgrade.
B
They
can
do
an
app
get
install
and
starting
to
add
in
the
tools
that
they
need
to
actually
go
and
do
data
exfiltration
or
explore
your
environment
and
move
laterally
or
horizontally.
And
so
how
do
you
in
this
environment?
Can
you
detect
abnormal
behavior
in
these
containers
that
are
very,
very
short-lived,
so
you
do
that
with
Falco's.
So
let
me
talk
about
the
architecture,
real,
quick.
What
we
do
is
we
take
different
pieces
of
information
and
we
run
them
through
the
Falco
rules
engine.
B
So
we
can
tap
into
system
calls
using
either
a
kernel
module,
which
is
our
kind
of
our
legacy
way
of
doing
things,
and
then
our
next
generation
way
of
doing
things
is
EBP
F
we're
actually
giving
away
books
Linux
observability
with
VPS
at
the
Cystic
booth.
The
author
of
the
book
is
giving
those
away.
B
So
EPF
is
an
area
that
we
do
a
lot
of
work
in
and
a
lot
of
study
in
as
well,
and
then
we
can
also
take
things
like
kubernetes
audit
events
and
then
kubernetes
metadata
combine
them
together
into
our
rules
engine
and
then
push
out
alerts
and
I'm
going
to
talk
about
the
BPF
instrumentation,
real,
quick.
So
the
way
that
this
works
is
essentially
you
deploy,
you
can
either
deploy
Falco
directly
on
your
kubernetes
nodes
or
your
hosts.
It
doesn't
need
to
be
running
kubernetes.
We
can
work
on
a
standard
Linux
system
as
well.
B
So
you
can
use
this
as
a
generic
host
intrusion
detection
system.
If
you
want
in
a
kubernetes
cluster,
what
you
typically
did
do
is
deploy
us
as
a
daemon
set.
The
EBP
up
program
gets
installed
on
that
nodes
kernel
and
that
allows
us
to
get
an
event
stream
of
all
the
system
calls
that
are
taking
place
and
going
through
the
system.
So
because
it's
a
shared
kernel
and
we've
installed
our
instrumentation
in
the
kernel.
We
can
then
see
everything
that's
happening
inside
of
your
containerized
workloads,
so
we
can
see
what
files
are
opened
up.
B
B
B
B
If
you
want
and
then
you
can
see
here,
we
have
a
macro
and
a
macro
is
essentially
a
shortcut
for
a
rule
condition,
so
it
just
makes
it
easier
to
write
something
over
and
over
and
over
again,
the
Falco
default
rule
set
provides
lots
of
macros
for
you.
So
if
you
wanted
to
catch
an
outbound
connection,
all
you
need
to
do
is
write
outbound
and
that
macro
is
available
to
you
and
it
will
detect
all
outbound
connections
automatically.
So
here
we're
gonna
say
our
directory
is
in
vendors.
B
So
basically
our
directory,
that's
set
in
the
system.
Call
information
is
one
of
these
four
and
so
what's
nice.
Here
is
what
I
decide
to
start.
Storing
binaries
and
opt
your
company
name
or
your
application
name
been
all
I
need
to
do,
is:
go
and
update
the
list
and
I
don't
need
to
update
my
macro
or
my
rule,
and
then
we
can
see
the
rule
here.
We
have
I'm
a
bender
and
event
direction.
Is
the
system
calls
returned,
that's
kind
of
not
important,
so
we'll
just
skip
it.
B
I've
opened
something
for
writing
in
this
directory
and
I'm,
not
a
package
management
process.
What
will
happen
is
this
Falco
will
alert,
you'll
get
the
username
you'll
get
the
command
and
the
file
name
of
what
was
actually
being
manipulated,
underneath
that
directory
makes
sense,
so
there's
a
whole
bunch
of
different
metadata
or
different.
What
we
call
field
classes
that
you
can
pull
in.
B
So
here's
another
rule
to
give
you
an
example
of
incorporating
that
metadata
information,
so
don't
run
this
in
production
because
you'll
get
a
lot
of
false
positives,
but
it's
good
for
an
example.
So
I've
spawned
a
process
with
using
the
exec
de
system
call
and
then
my
container
image
starts
with
node
and
my
proc
name
does
equal
node
so
basically
anytime,
that
a
container
runs
a
process
other
than
the
node
application.
We'll
get
an
alert
that
the
container
is
doing
something
abnormal.
B
We
have
a
bunch
of
different
rule
sets
if
you
go
to
security
hub,
dev,
I,
don't
have
a
slide
on
this,
so
this
is
the
only
time
I'm
gonna
say
its
security
hub
dev.
You
can
see
a
whole
bunch
of
different
Falco
rules
that
you
can
download
for
common
applications
and
one
of
the
things
that
we're
working
on
one
of
our
team
members
chris
nova
is
working
on
is
falco
CTL,
where
you
can
automatically
go
and
do
falco,
CTO
rule
install
and
you'll
be
able
to
download
those
rules
directly
from
security
hub.
B
So
another
area
that
we're
looking
for
contributions
and
hope
on
now.
We
can
also
look
at
kubernetes
audit
of
audit
events
as
well,
and
so
kubernetes
audit
logging
is
something
that
came
out
in
kubernetes
111,
it's
getting
more
and
more
mature
and
what
you
can
do
is
basically
get
a
chronological
record
of
every
single
change
that
goes
through
the
API
server.
B
So
any
time
that
someone
hits
the
API
server
you'll
get
four
different
options
that
you
can
choose
of,
whether
if
that
request,
just
came
in
or
whether
if
it
was
actually
completed
and
successful
and
that's
controlled
with
what's
called
audit
policy
and
then
once
you
have
this
audit
policy,
you
can
then
take
that
and
ship
that
off
to
some
other
location,
to
do
processing
on
it.
What
we'll
do
is
we'll
actually
apply
the
Falco
rule
set
to
this
information
and
the
different
ways
that
you
can
configure.
This
is
log
files
web
hooks.
B
These
two
methods
aren't
available
if
you're,
using
something
like
GK
or
a
KS
or
a
managed
kubernetes
service,
where
you
don't
have
access
to
the
master.
This
has
changed
on
the
master,
so
what
the
kubernetes
community
did
is
they
created
something
called
audit
syncs,
where
you
can
actually
dynamically
control
this
by
putting
a
JSON
document
up
into
the
career
day's
API
server?
So
here's
what
an
audit
event
looks
like
and
in
this
audit
event
you
can
see
that
the
response
was
complete.
B
B
The
other
interesting
thing
is:
is
that
if
kubernetes
ever
changes,
what
this
JSON
document
looks
like
will
still
take
it
and
parse
it,
and
we
can
actually
pull
any
value
out
by
just
calling
j
event
value
and
the
json
pointer.
This
also
allows
us
to
implement
new
input
sources
as
well.
So
after
we
get
done
with
the
outputs
api,
we're
gonna
be
working
on
the
input
api
to
allow
you
to
have
more
sources
where
you
can
actually
write
these
rules
and
extend
the
Falco
language
for
detecting
things.
B
B
We
can
see
we
have
the
AWS
access
key,
so
we're
just
looking
for
these
strings,
and
while
this
is,
this
is
not
gonna
stop
state-level
actors
right,
but
normally
your
breaches
and
things
like
that
happen
because
Bob
in
accounting,
you
know
made
an
s3
bucket
public
or
Bob,
and
accounting
went
and
no
offense
to
Bob
he's
just
trying
to
do
his
job
but
went
and
posted
things
and
a
github
repository.
They
were
not
supposed
to
or
unencrypted
secrets
and
other
things
like
that
right.
B
So
we
now
have
a
macro
called
the
config
Maps:
okay,
a
target
resource,
config
Maps,
meaning
that
we're
can
dip.
You
lating
the
config
map
through
the
API,
we're
doing
a
create,
update
or
patch.
So
any
one
of
these
actions,
and
then
we
have
the
rule
itself,
so
I'm
a
config
map.
Somebody
tried
to
modify
me
and
we
contain
private
credentials.
So
what
will
happen
is
is
we'll
get
the
alert
that
a
config
map
was
created
with
private
credentials,
we'll
get
who
did
it?
The
other
interesting
thing
is:
is
we'll
also
get
the
entire.
B
Configured
in
there
we've
now
leaked
your
secret
and
unis
we're
forcing
you
to
rotate
it.
We
should
probably
awfully
scape
that
that's
another
area
of
opportunity.
If
somebody
wants
to
contribute
alright,
so
I'm
going
to
talk
about
installing
real
quick
in
the
kubernetes
world,
we
can
install
via
helm.
We
also
have
Linux
packages
as
well.
B
Helm
is
one
of
the
preferred
ways,
lots
of
configuration
options
and
then
we're
also
working
on
Falco
CTL
install
as
well
that
will
install
a
coup
falco
into
your
kubernetes
cluster
as
well.
So
let's
talk
about
how
we
use
Falco
and
I'll
demo,
this
real
quick.
So
we
wrote
something
called
the
kubernetes
response
engine
with
security
playbooks.
What
it
does
is
it
allows
you
to
react
to
these
events.
So
we
have
a
number
of
different
play
books
you
can
kill
within
an
offending
pod.
You
can
taint
nodes
to
prevent
scheduling.
B
You
can
isolate
a
pod
with
networking
policy
and
so
forth,
so
that
then,
when
this
container
goes
rogue
you
can
keep
it
around
but
isolate
it
and
then
go
in
and
do
forensics
on
it
as
well.
You
can
subscribe
to
different
level
of
alerts.
You
can
subscribe
to
individual
alerts,
and
so
what
this
allows
you
to
do
is
you
can
have
different
actions
that
take
place
depending
upon
the
alert
and
severity.
B
So
what
this
looks
like
is
is
that
we
publish
into
Nats
and
then
from
Nats
boobless,
picks
it
up
and
runs
a
small
function
and
then
we'll
take
action
on
the
particular
function
or
on
the
particular
finding
container.
We
have
the
same
thing
for
AWS.
We
have
the
same
thing
for
GC
P
as
well,
so
this
is
another
area
to
contribute,
we're
written
in
C++
and
then
we're
starting
to
write
more
stuff
and
go,
and
so,
if
you're
not
comfortable
with
C++,
we
have
a
lot
of
gamal
that
you
can
write.
C
B
B
I
can
see
that
there's
been
some
events
that
have
happened.
We
can
see
that
a
terminal
respond
in
a
container
ignore
this,
because
I
don't
have
this
tuned
for
this
particular
machine.
You
can
see
that
the
bash
history
was
deleted
and
other
information
like
that
as
well.
You
notice
that
this
is
in
JSON,
so
it's
programmatic,
so
you're
able
to
parse
it
and
send
it
into
another
system.
So
what
I'm
going
to
do
here
is,
let
me
just
do
a
coop
CTO.
C
B
B
And
I'm
just
gonna
get
in
this
container
to
start,
manipulating
it
and
changing
things
and
doing
stuff
with
it
and
I
can't,
and
the
reason
why
I
can't
is
because
what's
happened
is
is
immediately
within
a
matter
of
milliseconds.
That
function
was
triggered
and
it
kicked
me
off
of
the
container
and
it
didn't.
Not
only
did
it
kick
me
off
the
container,
it
actually
deleted
the
container
and
you
can
see
that
that
containers
went
away
and
a
new
one's
been
been
deployed.
B
So
this
is
what
you
can
do
with
Falco,
and
we
really
think
that
it's
important
that
you
need
to
look
at
this
entire
was
you're
looking
at
your
stub
ops,
dub,
suck
ops
security
posture.
Looking
at
this
entire
workflow,
looking
at
your
dub
suck
Ops
pipeline
and
templating
it
and
making
sure
that
you're
putting
all
the
layers
in
place,
but
then
also
it's
important
to
actually
watch
what's
happening
in
production
so
that
you
can
then,
because
everything
that
happens
in
production
all
of
this
has
to
feed
back
into
your
development
cycle.
B
So
the
reason
why
that
yellow
line
ends
up
continuing
on
is
because,
whatever
you
discover
and
whatever
malicious
activity
you
might
discover
then
goes
into
patching
something
in
your
environment
and
working
through
that
dub
psych,
ops
lifecycle.
So
if
you
have
the
jenkins
templating
engine,
if
you're
able
to
actually
more
easily
go
and
incorporate
those
changes
in
and
those
feedback
loops
in
the
faster
that
you
can
move
overall
in
your
pipelines.
So
with
that,
thank
you
very
much.
These
are
QR
codes
to
our
documentation.