►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Intro to Falco: Intrusion Detection for Containers - Shane Lawrence, Shopify
How do you protect Kubernetes clusters from malicious behavior? Role-based access control won't stop a user who's authorized to create pods from deploying hundreds of coin miners, and Intrusion Detection Systems at a network edge won't catch requests from a compromised container to the API server. Falco joined CNCF Incubator as an open-source runtime monitoring tool that combines kernel-level visibility with cluster-level awareness, making it possible to implement security policy and assert if these policies have been violated. In this session, Shane will demonstrate detection use cases, and discuss how Shopify has been using Falco since 2018 to monitor containers in a cloud environment that processes $100 million+ per day. Attendees will learn how to deploy Falco at scale, implement and change the ruleset, avoid common pitfalls with eBPF probes and kernel modules, and manage alert volume.
https://sched.co/Zewd
A
But
first
I
have
a
confession:
I'm
not
actually
a
falco
maintainer,
and
this
is
a
maintainer
track
talk,
I'm
barely
even
a
contributor,
but
they
thought
it
would
be
cool
to
have
an
end
user
go
through
a
few
things
and
I've
been
using
it
in
one
form
or
another
since
2018.
So
a
little
bit
about
myself,
I'm
on
the
infrastructure
security
team
at
shopify.
A
I've
worked
in
the
past,
mostly
with
intrusion,
detection
systems
like
network
intrusion,
detection
or
nids,
using
a
tool
called
snort,
also
a
bit
of
circada,
the
zeek,
formerly
known
as
bro,
and
also
some
host
ids
or
hids
like
os
sec.
I've
also
done
some
sim
or
security
information
event,
management
and
logs,
which
is
basically
gathering
a
haystack
of
events
from
all
the
things,
aggregating
them
and
trying
to
pick
out
malicious
needles
and
all
of
it
that's
hard
because
most
of
the
things
that
feel
like
needles
turn
out
to
just
be
pointy
pieces
of
hay.
A
Now
I
spend
most
of
my
time
doing
none
of
those
things
I
joined
shopify
in
2017
on
the
infrastructure
security
team.
Since
then,
my
primary
focus
has
been
on
preventing
anything
bad
from
happening
in
the
first
place,
mostly
by
giving
developers
paved
roads
and
guardrails,
so
that
it's
easier
for
them
to
deploy
secure
applications
than
it
is
to
deploy
insecure
ones.
A
A
So
this
is
what
people
see
when
they
see
shopify,
and
this
will
get
into
why
shopify
is
using
it.
So
we've
got
over
a
million
stores
on
shopify.
They
all
have
control
over
their
own
look
and
feel
to
the
vast
majority
of
people
who
will
interact
with
shopify
in
some
way,
they'll
see
something
that
looks
a
little
bit
like
this
and
most
of
the
time
they
don't
even
realize
that
they're
looking
at
a
shopify
store.
A
So
that's
what
our
customers
see,
but
this
is
what
our
merchants
see.
They
see
that
they're
dealing
with
shopify
they've
chosen
shopify.
They
know
it's
us.
Big
businesses
have
chosen
us
because
they
want
a
reliable
platform
that
leaves
the
branding
to
them,
and
small
businesses
want
it
to
be
easy
to
benefit
from
that
same
reliable
platform
as
large
ones.
A
This
is
what
I
see,
though
I
see
10
000
services
on
50
clusters
and
that's
a
lot
of
infrastructure
to
secure
so
in
some
of
these
clusters,
including
the
ones
that
actually
process
the
payment
card
data
for
190
million
dollars
a
day.
We
need
to
do
everything
we
can
to
ensure
that
we're
preventing
any
unauthorized
access.
We
do
this
by
following
best
practices
and
by
deploying
scripts
to
scan
and
ensure
that
all
of
our
workloads
really
are
following
the
things
we've
learned
in
the
past.
A
Bounty
I
mentioned
this
in
my
other
talk,
but
if
you
missed
any
of
the
things
on
the
left,
then
you
might
be
vulnerable
and
if
an
attacker
knew
about
any
of
the
things
on
the
right
before
you
had
a
patch
for
it,
then
you
were
probably
vulnerable
and
if
you're
vulnerable,
you
might
be
compromised.
We
need
monitoring
in
order
to
be
sure
that
everything
else
we're
doing
is
working
and
sometimes
we're
just
not
even
aware
of
the
things
that
could
lead
to
a
compromise.
This
is
why
effective
monitoring
is
so
important.
A
Now
we've
chosen
falco
specifically
because
it
sees
every
syscall.
It
knows
exactly
what
is
happening
on
the
system
at
the
kernel
level,
but
it's
also
containers
and
kubernetes
aware.
So
in
our
case
we
use
docker,
so
it
pulls
the
docker
socket
to
see
exactly
which
container
correlates
to
the
process
that
it
saws
this
call
from,
and
then
it
looks
up
from
the
kubernetes
api.
What
pod
and
name
space
corresponds
to
that.
A
A
It
uses
ebpf,
which
is
extended,
berkeley,
packet
filters
and
that's
really
cool,
because
for
one
thing
it's
performant
and
another.
It's
it's
read
only
at
the
user
space
level,
so
it's
a
little
bit
safer
than
using
some
of
the
other
drivers
that
might
be
available,
also
focus
open
source.
It's
under
active
development.
It's
got
a
very
responsive
development
team
and
if
you
need
to
ensure
a
quick
response
from
certain
members
of
their
team
in
italy,
I've
heard
they
love
pineapple
pizza,
so
just
lead
with
that.
A
A
A
Tooling
validates
the
rules
quickly
before
it
actually
builds
this,
and
this
is
actually
really
important
to
us,
because
if
you
get
a
typo
or
something
something's
wrong
with
your
syntax,
you
want
to
know
that
right
away
before
you've
even
pushed
this
image,
so
that
you
can
find
out
before
you've
gone
to
all
the
trouble
of
deploying
this
to
a
hundred
nodes
in
a
cluster.
Even
if
that
is
just
a
staging
cluster,
you're
saving
a
lot
of
time,
most
kernels
actually
have
a
driver
pre-built
for
you.
A
But
if
your
cloud
provider
has
tweaked
the
kernel
for
your
image
and
it's
now
non-standard,
then
you
might
need
to
build
the
driver
yourself.
So
have
your
tooling
do
that
as
a
separate
step
in
a
separate
docker
stage,
and
then,
when
you've
already
got
this
image,
you
can
have
it
already
populated
and
you
don't
need
to
wait
for
it
to
build
every
time.
The
container
starts
up
because
it
takes
a
couple
of
minutes.
A
A
A
So
in
production
you
don't
want
to
have
to
build
a
new
image
and
wait
for
the
whole
thing
to
deploy
every
time.
The
rules
change
so
use
a
configmap.
Instead,
that's
what
they're,
for
you
put
the
configuration
files
in
a
config
map
the
then
you
project
that
config
map
as
a
volume
and
mount
it
as
etsy
falco,
and
it
turns
out
actually
that
the
helm
chart
already
does
this.
If
you
modify
the
config
map,
you
can
see
it
in
the
falcon
name
space.
A
Then
it
will
change
the
contents
of
etsy
falco
where
the
config
is
stored,
but
the
problem
with
the
helm
deployment
is
that
there's
nothing
there
to
detect
that
and
and
restart
falco,
or
get
it
to
recognize
that
the
rules
have
changed.
So
the
only
way
you
can
do
this
is
with
helm
itself
and
and
that
restarts
everything
or
you
can
send
a
hang-up
signal.
A
The
the
better
alternative,
I
think,
is
you
edit
the
docker
file
to
run
a
script
in
each
falco
container,
and
then
you
tell
that
falco
container
to
run
I
notify
weight
and
watch
to
see
if
anything
in
this
directory
changes,
then,
when
you
change
the
config
map,
it
changes
the
etsy
falco
directory
and
automatically
sends
a
kill,
sig
hup
to
the
falco
process.
This
hang
up
signal
will
tell
it
to
just
reload
the
rules
or
the
configuration
without
any
downtime
and
an
easy
way
to
do.
A
A
So
we
got
into
a
bit
of
a
panic.
Had
someone
taken
over
the
cluster
was
one
of
our
upstream
dependencies
compromised?
We
looked
at
the
alerts
more
closely
and
saw
that
the
traffic
was
coming
from
falco
itself.
So
now
we're
wondering
is
falco
compromised.
Well,
no,
it
turns
out
that
the
update
had
added
a
rule
that
detects
traffic
to
coin
mining
websites,
and
it
knows
this
by
their
domain
name.
So
when
we
upgraded
falco,
it
started
up,
saw
the
rule.
A
That
said,
tell
me
if
you
see
anything
talking
to
the
mining
site
and
on
each
node
falco
dutifully
performed
a
dns
lookup
for
the
mining
site,
so
that
it
would
know
exactly
where
traffic
should
not
go.
The
rule
that
the
falco
team
had
created
wouldn't
trigger
itself,
but
we
actually
had
another
rule
that
was
sort
of
an
extra
defense
against
data
exfiltration
and
falco's
own
dns.
Lookups
were
picked
up
as
these
unexpected
outbound
connections
to
the
mining
site
and
our
data
exfiltration
rules
went
crazy.
A
Now
in
this
example
problem.
This
is
one
of
the
first
things
you
might
see
is
that
when
falco
launches
it
detects
this
privileged
container
launch,
and
so
we
need
to
teach
it
that
this
is
okay
and
the
solution
to
this.
This
is
just
a
sample
of
the
alert
that
you
would
see,
and
then
this
is
what
you
need
to
add
to
the
rules
file
in
order
to
teach
falco
that
this
is
okay.
A
So
in
our
case,
there's
an
image
repository
called
falco
security
falco
and
the
list
itself
had
prefixed
this
with
docker,
and
so
it
wasn't
recognizing
this
as
a
falco
container
was
an
understanding
that
it's
okay
to
start.
By
doing
this,
I
mean
you
get
rid
of
all
the
noise
that
you
would
see
when
you
first
start
falco.
A
This
really
isn't
that
big
of
a
deal
we're
okay
with
this,
but
because
the
falco
developers
didn't
expect
really
any
of
these.
They
created
this
so
that
it
would
appear
as
a
critical
alert,
and
this
actually
is
set
up
by
default,
to
report
this
as
a
critical
alert
every
30
seconds
on
every
node
and
and
so
if
this
is
happening
all
the
time,
that's
going
to
create
so
much
fatigue
so
fast
for
no
reason
at
all.
A
A
So
these
are
a
few
pretty
simple,
straightforward
things
that
you
can
do
when
you're
starting
out
that's
going
to
make
the
experience
a
lot
better
for
you
when
you're
trying
to
use
falco,
but
it
still
doesn't
give
you
a
whole
lot
of
usefulness.
So,
let's
take
a
look
at
some
of
the
things
that
we
can
do
and
actually
in
order
to
actually
use
falco
and
detect
some
attacks.
A
So
if
we've
got,
for
instance,
suspicious
shell
access
in
a
container
now,
if
you
have
sensitive
workloads,
hopefully
you
aren't
routinely
keep
control
executing
into
them
to
make
changes
because
they're
supposed
to
be
containerized.
After
all,
I'm
going
to
demonstrate
someone
getting
shell
access
and
I'm
just
going
to
use
cubecontrol
for
the
example,
but
an
attacker
might
also
be
able
to
do
this
by
exploiting
some
other
vulnerability
in
that
container
or
perhaps
by
finding
some
less
important,
but
also
less
secure
workload
and
then
pivoting
to
this
one.
A
We're
switching
to
the
falco
namespace
and
I'm
going
to
look
at
which
pods
are
running
here
and
let's
just
pick
the
first
one
and
we're
going
to
suppose
in
this.
You
know
contrived
example
that
the
attacker
has
somehow
compromised
this
falco
pod,
so
our
defenses
have
failed,
but
we
can
still
detect
it.
A
So
what
might
an
attacker
do
when
they
get
in
here,
they're,
probably
going
to
look
at
some
things,
and
then
maybe
they
want
to
sniff
traffic
for
some
reason
to
see
what
information
is
going
through
here
or
perhaps
to
do
a
little
bit
of
surveillance,
network
reconnaissance.
So,
as
the
attacker
I
might
decide,
I
want
to
install
tcp
dump
now,
for
the
sake
of
the
example,
I
think
I've
I've
shown
enough.
Maybe
I've
had
a
change
of
heart
and
I
really
don't
want
to
fall,
so
I'm
just
going
to
exit
the
container
there.
A
Let's
see
what
it
actually
looks
like
when
falco
detects
this,
so
we're
going
to
take
a
look
at
falco
pod
and
see
what
it
found
so
down
here
at
the
very
bottom.
You
can
see
that
it
detected
two
things,
and
these
are
really
important.
Rules
to
monitor
a
shell
was
spawned
in
a
container
with
an
attached
terminal.
This
is
important.
A
You
want
an
alert
when
something
like
this
happens,
because
again
it
shouldn't
be
normal
practice
for
people
to
just
cube
control
exec
into
these
things,
but
even
more
importantly,
is
this
package
management
process
launched
in
a
container
that
should
really
already
have
all
of
its
packages
installed.
So
when
you
see
something
like
this,
you
can
see
the
command
I
have
to
get
installed,
tcp
dump
by
user
root
and
as
soon
as
a
human
looks
at
this
they're
going
to
know
that
something
strange
is
going
on
and
they're
going
to
be
able
to
respond
to
that.
A
So
another
potential
use
case
that
you
might
find
valuable
is
related
to
the
instance
metadata
service
and
in
2018
I
gave
a
talk
about
a
report
that
we
received
from
a
researcher
who'd
managed
to
exploit
a
vulnerable
application
that
we
were
running
in
a
second
tier
environment
at
shopify,
and
he
used
a
combination
of
things
to
get
into
this
metadata
service
and
ultimately
got
access
to
these
cuban
secrets
that
could
have
potentially
led
to
a
cluster
takeover.
So
I've
mentioned
before
that.
A
It's
way
more
important
to
prevent
access
to
these
things
than
it
is
to
monitor
your
career,
ending
in
real
time.
So
in
my
demo,
cluster
I've
got
workload,
identity
enabled
and
what
that
does
is
puts
a
proxy
out
in
front
of
the
metadata
service
and
it
hands
out
only
the
appropriate
service
account
token
and
it
blocks
any
inappropriate
requests.
So
they
can't
go
through.
So
I'm
going
to
pop
into
a
normal
container
and
I'm
going
to
hit
that
metadata
instance
and
I'm
going
to
see
what
it
shows
me.
A
A
A
A
A
Yeah,
okay,
so
we
can
see
we
can
only
get
the
cluster
location,
the
cluster
name
and
the
cluster
id.
These
are
things
that
it's
pretty
normal
to
want
to
be
able
to
access
when
you
are
an
application
running
in
the
cloud
somewhere.
So
this
is
actually
pretty
good.
It's
good
to
see
this
and
then,
if
we
also
want
to
I'm
going
to
copy
and
paste
this
time,
so
I
don't
make
any
mistakes.
A
A
A
So
we
want
to
do
the
same
thing
and
we
want
to
see
what
attributes
are
available
again,
this
one's
on
the
host
network.
So
what
this
pod
can
do
is
it
uses
the
same
ip
address
as
the
host
itself
and
shares
a
network
with
it,
and
this
might
be,
for
instance,
an
infrastructure
workload
that
needs
to
be
able
to
interact
with
everything
running
on
that
particular
node.
But
if
we
do
gain
access
to
this,
what
we
see
here
is
that
there's
a
lot
more
information
available.
A
The
reason
is
it's
on
the
host
zone
network,
so
it
doesn't
go
through
the
proxy.
It
goes
directly
to
the
instance
metadata
service
and
what
you
can
do
with
this
kind
of
access
is
exactly
what
the
researcher
did
in
the
talk
that
I
gave
a
couple
years
ago.
You
get
the
cube
end
and
then
you've
got
the
cubelet
cert
and
the
cubelet
key.
Here
you
can
take
those
and
then
you
can
use
them
to
impersonate
the
cubelet
bootstrap
user,
which
you
know
if
other
protective
measures
aren't
followed,
could
potentially
lead
to
a
cluster
takeover.
A
A
Plural,
this
is
not
an
application
service
account.
This
is
actually
the
instance's
own
compute
service
account
that
the
vm
itself
runs
as
so.
If
the
attacker-
and
this
example
me
takes
this,
we
can
get
the
token
from
that
and
we
can
use
that
to
impersonate
this
vm
anywhere
on
the
cloud
provider's
infrastructure.
So
take
that
pull
the
token
and
there
it
is.
We
could
use
that
and
impersonate
the
vm
itself,
which
again
is
not
desirable.
A
A
And
you
can
see
here
that
they
did
in
fact
detect
it.
You
can
see
outbound
connection
to
cloud
instance
metadata
service,
and
it
shows
the
exact
command
that
I
used
here.
Trying
to
get
the
service
account
trying
to
get
the
cube
end,
and
so,
if
we
had
been
running
falco
with
this
rule,
enabled
in
that
second
tier
cluster
in
2018,
then
we
would
have
immediately
gotten
alert
when
that
researcher
was
able
to
compromise
this,
and
it
would
have
saved
us
25
000
in
that
case.
A
So
what
else
can
we
do?
I'm
actually
not
going
to
demonstrate
this
one,
because
it's
going
to
look
pretty
much
the
same,
but
what
we
can
do
is
you
don't
even
need
to
get
a
host
network
pod.
If
you
can
get
a
privileged
pod,
then
you'll
be
able
to
do
the
exact
same
thing,
because
even
though
just
running
it
by
default,
it
would
still
go
through
that
proxy.
It
would
still
hit
workload
identity.
A
If
you
have
root
privileges,
you
can
change
the
rules.
The
redirect
to
the
proxy
is
done
with
the
network
fabric,
and
so,
in
our
case
we're
running
iptables
with
calico.
So
an
iptables
rule
tells
requests
to
metadata.google.internal
or
169254,
as
you
saw,
it
tells
all
those
to
go
to
the
proxy
instead.
But
if
you
have
root
privileges,
you
just
change
the
rules,
you
don't
go
to
the
proxy.
You
go
direct
to
the
instance
metadata
and
you
get
the
same
pwnage
that
we
saw
in
the
last
example.
A
So
there's
a
lot
you
can
do
and
in
order
to
be
able
to
detect
these
things,
it
really
fills
in
the
gaps.
And
so,
if
you've
missed
anything,
then
you
want
this
ability
to
see
what's
happening
in
your
clusters.
So
I'll
go
to
another
example.
Now
it's
been
a
strange
summer,
there
are
a
lot
of
people
outside
who
are
away
from
each
other
and
there
are
some
people
who
are
inside
away
from
each
other,
and
some
of
those
people
who
are
inside
found
some
exciting
new
vulnerabilities
in
kubernetes,
while
they
were
at
home.
A
It
just
fills
up
a
file
all
the
way
to
a
hundred
percent
with
junk,
and
once
that
volume
is
full.
That
would
be
bad
enough
in
a
container's
own
file
system.
But
the
problem
is
it:
does
this
with
etsy
hosts
which
lives
on
the
node,
not
the
container,
so
it
takes
down
the
whole
node,
not
just
the
pod,
and
when
this
came
out,
the
cubelet's
eviction
manager
forgot
to
consider
it
c
hosts.
A
So
it
would
allow
it
to
just
keep
filling
up
until
the
node
failed
and
if
you've
got
large
nodes
that
could
run
many
different
workloads.
It
could
mean
this
vulnerable
application
doesn't
just
go
down
itself.
It
brings
down
100
larger,
more
important
ones,
so
this
is
actually
detectable
by
falco.
Just
using
the
default
rule
set,
so
I'm
going
to
demonstrate
that
now
so,
let's
go
into
for
the
sake
of
time,
we're
just
going
to
keep
control
exec
again.
A
This
could
be
some
other
exploit
that
someone's
used
to
gain
access
to
this
pod.
They've
compromised
it
in
some
way,
and
you
want
to
be
able
to
contain
that.
But
thanks
to
the
cv,
they
can
bring
the
whole
node
down.
So
I'm
in
this
privileged
pod
and
for
the
example
I'm
going
to
do,
is
I'm
going
to
write
one
line
here
and
I'm
going
to
put
it
at
the
end
of
etsy
hosts
so
in
in
the
real
world.
A
If
someone
was
taking
advantage
of
this,
they
would
be
writing
a
huge
amount,
not
just
this
one
line,
but
this
suffices,
for
example.
So
let's
take
a
look
at
what
falco
detected
when
we
decided
to
do
this
so
again,
we'll
take
a
look
at
our
logs
from
our
falco
pods
and
look
for
this
rule
triggering
and,
let's
just
say
for
the
last
five
minutes
there.
It
is
so
at
the
exact
time
that
I
ran
this
command.
You
can
see
error
file
below
etsy
opened.
For
writing.
This
is
just
the
default
rule.
A
If
you
wanted,
you
could
make
it
more
specific
to
detect
the
cve
specifically,
but
it's
almost
better
in
my
opinion,
that
it
just
detects
this
out
of
the
box.
You
teach
falco
what
should
be
able
to
write
below
this
directory
and
if
other
things
are
doing
it,
then
we
know
that's
bad
and
if
you're
looking
closely,
you
might
notice
that
it
didn't
show
the
echo
command.
A
That's
just
because
echo
is
built
in
so
the
binary
itself
was
just
shell,
but
you
can
see
which
file
it
was
that
was
modified,
and
then
you
can
use
some
other
method
to
decide
how
you
want
to
this
maybe
go
check
the
file
size
of
etsy
hosts
and
figure
out
if
you've
been
compromised
or
if
you
just
need
to
allow
list
whatever
it
is.
That
did
this.
A
So
these
are
some
examples
that
we've
looked
at
of
use
cases.
There
are
hundreds
more
that
you
can
find
in
the
falco
default
rule
set
also
in
falco.org.
You
can
find
out
a
lot
more,
but
let's
take
a
look
at
what
I
think
is
a
more
difficult
problem
and
that
is
managing
all
of
these
alerts.
So
you
need
to
make
sure
that
you're
handling
your
output.
A
It
needs
to
be
visible,
searchable,
aggregated
and
annotated
so
that
you
can
see
what
happened
and
when
so,
you
can
see
which
of
your
alerts
are
being
problematic
and
which
ones,
maybe
are
just
fine
the
way
they
are,
and
you
also
need
to
be
able
to
add
annotations
to
these
things.
That's
really
important
because
you
don't
want
to
spend
time
investigating
something
that
someone
else
on
your
team
is
already
looking
at
and
has
maybe
already
resolved.
A
Also
by
being
able
to
see
when
this
alert
has
triggered
before
you
can
decide.
If
maybe
it's
going
off
too
often
and
instead
of
alerting
on
some
specifics,
maybe
you
can
make
the
rule
a
little
bit
more
broad
or
go
the
other
way,
make
it
more
specific,
so
teach
falco,
what's
normal
in
your
environment,
so
that
you're
not
getting
false
alarms
all
the
time.
This
is
going
to
create
alert
fatigue.
If
it's
going
off
too
much
and
a
big
part
of
handling,
this
is
normalization.
A
So
as
soon
as
you
see
something-
and
you
know
that
it's
a
false
alarm-
you
need
to
promptly
go
in
and
allow
this
this.
You
should
also
make
it
as
simple
as
possible,
maybe
using
pr
templates
or
even
automation,
so
that
when
a
person
identifies
that
this
doesn't
need
to
bother
somebody.
This
doesn't
need
to
wake
somebody
up
in
the
morning.
They
can
very
quickly
and
easily
assign
that
combination
of
pod
action,
container
and
other
information
to
the
allow
list
now,
once
you've
got
all
of
that
handled
out.
A
If
you
do
run
into
problems,
you
can
reach
out
to
falco
on
the
kubernetes
slack
again.
One
of
the
things
that
we
like
most
about
falco
is
how
responsive
the
team
is,
if
you're
not
sure,
if
something's
a
bug
or
you've
just
done
something
wrong,
then
falco
slack
is
the
place
to
go.
They
also
have
a
weekly
community
call.
If
you
know
that
it's
a
bug,
you
can
open
a
github
issue
or
if
you
really
want
to
contribute.
The
best
thing
you
can
do
is
open
a
pull
request
and
give
back.
A
So
thanks
so
much
for
watching
my
talk.
If
you
do
want
to
get
involved,
it's
a
cncf
project,
there's
so
many
ways
you
can
help
out.
Also
you
can
learn
more
about
me
see
what
falco
is
all
about
jump
into
the
slack
channel
there
or
also
shopify,
is
now
digital
by
default
and
we're
hiring.
You
can
also
check
out
the
shopify
engineering
blog
to
see
what
else
we're
doing
so,
thanks
again,.
A
Okay,
so
I'll
take
any
questions
that
you
have
now
and
I
see
we've
got
a
question
already,
so
the
question
was:
is
it
possible
to
automatically
take
actions
and
with
falco
by
itself?
No,
but
there
are
so
many
ways
that
you
can
consume
these
streams
that
you
can
use
something
as
trivial
as
a
shell
script.
That's
monitoring
a
log
file
to
do
this,
or
I
mean,
if
you
have
an
advanced
sim,
then
really
the
sky's
the
limit.
A
There
are
so
many
options
that
you
have
for
going
back
and
changing
things
and,
like
I
mentioned
with
the
config
map
being
able
to
very
quickly
deploy
these
rules,
changes
is
going
to
be
critical
to
having
any
kind
of
automation
involved
at
all
and
in
addition,
it's
honoring,
and
when
do
you
automatically
take
action?
I
think
that's
extremely
case
by
case.
A
I
would
want
to
be
absolutely
certain
that
the
action
was
the
right
one
before
I
had
automation,
doing
something
that
could
affect
a
very
critical
production
environment
but
again
like
this
is
a
tool
that
we're
using
for
detection
right
now.
If
you
want
to
get
into
prevention
or
taking
automatic
action,
then
that's
a
bit
more
advanced
and
it's
something
that
falco
will
allow
with
its
capabilities.
A
But
it's
not
something
that
it
does
by
itself
and
as
you
mentioned
yes,
because
any
false
positive
would
disrupt
your
own
service.
So
that's
that's
why
you
know
you
want
to
be
very
careful
with
the
sort
of
automation
that
you
would
implement
here
and
the
next
question
is:
is
there
a
free
edition
of
falco,
or
is
this
a
premium
feature?
So
falco
is
completely
open
source.
It
was
donated
by
systig
to
the
cncf.
A
A
Falco
itself
does
not
have
a
fancy
web
front
end
or
anything,
and
I
actually
kind
of
like
that,
because
then
I
can
just
integrate
it
into
the
tools
that
I
was
using
anyway,
and
if
I
want
to
check
something,
I
showed
in
all
of
these
examples
just
looking
at
the
standard
output
of
falco.
It's
also,
you
know,
that's
probably
not
how
you're
going
to
consume
it.
So
we
have
fluent
d
ingesting
everything
from
standard
out
on
all
of
our
docker
containers
and
dumping
it
into
splunk.
A
So
that's
one
way
of
consuming
it,
there's
also
falco
sidekick,
which
makes
it
really
easy
to
send
alerts
to
various
destinations,
such
as
slack,
and
you
might
have
seen
because
my
the
slides
that
were
supposed
to
not
show
did
show.
So
there
were
a
couple
times
where
you
might
have
seen
a
slack
alert
pop
up.
While
I
was
demonstrating
something
and
that
is
falco's
sidekick.
Sending
to
slack
the
problem
that
I
find
with
using
slack
for
alerts
is
that
it
becomes
extremely
overwhelming
if
an
alert
triggers
100
times.
A
I
really
only
want
to
know
once
I
want
one
alert
that
says
this
triggered
100
times
and
there's
no
real
way
to
aggregate
consolidate
those
in
slack.
So
you
can
just
get
flooded
with
a
spammy
alert
and
you
might
miss
a
much
higher
priority
that
came
through
so
that
can
be
really
useful
in
the
beginning.
But
as
you
try
to
wrangle
all
of
these
many
alerts,
you
are
probably
going
to
want
to
have
something
a
bit
more
sophisticated
and
a
bit
more
searchable.
A
Oh,
this
is
a
really
good
question.
Could
you
walk
through
how
you
test
the
rules
before
deploying
them?
So
we
have
a
shell
script
that
does
a
very
simple
verific
verification
check
on
this
before
it
gets
deployed
and
that's
important
because
we
don't
want
to
waste
all
this
time
for
ci
to
try
to
deploy
a
whole
new
container
only
for
it
to
fail
at
the
last
step
because
of
a
syntax
error,
but
actually
making
sure
that
the
rules
are
working
as
intended.
A
We
use
a
staging
environment
for
so
we
can
trigger
some
of
these
alerts
on
command
or
by
contriving
some
sort
of
action
just
like
they
did
in
the
demo.
And
if
we're
experimenting
with
a
new
rule,
then
we'll
come
up
with
how
we
want
it
to
work.
We
might
spin
it
up
in
a
sandbox
and
then,
when
we
think
we've
got
it
the
way
we
want
it,
we'll
run
it
in
staging
and
test
it
to
make
sure
that
it
works
correctly.