►
From YouTube: Container Forensics: What to Do When Your Cluster is a Cluster - Maya Kaczorowski & Ann Wallace
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Container Forensics: What to Do When Your Cluster is a Cluster - Maya Kaczorowski & Ann Wallace, Google
When responding to an incident in your containers, you don’t necessarily have the same tools at your disposal that you do with VMs - and so your incident investigation process and forensics are different. In a best case scenario, you have access to application logs, orchestrator logs, node snapshots, and more. In this talk, we’ll go over where to get information about what’s happening in your cluster, including logs and open source tools you can install, and how to tie this information together to get a better idea of what’s happening in your infrastructure. Armed with this info, we’ll review the common mitigation options such as to alert, isolate, pause, restart, or kill a container. For common types of container attacks, we'll discuss what options are best and why. Lastly, we’ll talk about restoring services after an incident, and the best steps to take to prevent the next one.
https://sched.co/MPdK
B
Stephen
just
said:
I'm
Micah
Trotsky,
my
product
manager
and
security
at
Google.
So
what
we're
going
to
be
talking
about
today
is
how
to
get
started
with
container
forensics
when
it
turns
out
that
your
cluster
is
a
cluster
we'll
cover
why
you
should
care,
because
the
tactic
containers
are
real,
what
forensics
are
and
how
to
prepare
ahead
of
time.
Data
sources
that
you
can
use
for
investigations
for
containers,
specifically
common
mitigation
options
and
then,
finally,
what
you
can
do
today
so
talking
about.
Why
are
we
even
talking
about
container
forensics?
B
Do
containers
actually
get
attacked
at
all,
and
so
what
we're
looking
back
at
last
year?
What
are
the
attacks
that
we've
seen
in
the
wild
in
February
2018
Tesla
made
the
news
for
attackers
finding
credentials
that
they
had
gained
to
mind
cryptocurrency
in
May
Shopify
had
a
researcher
report
of
ulnar
ability
in
their
cluster
configuration.
B
Thankfully,
this
was
not
exploited
in
June
weight
watchers
face
the
exact
same
issue
as
Tesla
with
an
unsecured
dashboard
and
the
same
month,
several
container
images
were
found
on
docker
hub
that
had
embedded
malware
for
mining
cryptocurrency
more
recently
in
late
April
20
1980.
A
hub
database
which
held
user
data
including
docker,
hub
usernames,
hash,
passwords
and
github,
and
bucket
tokens
used
for
auto
building
docker
container
images.
B
Now
this
affected
more
than
190,000
accounts,
but
an
unknown
number
of
images,
and
if
any
of
the
midges
that
were
affected,
how
many
times
those
were
pulled-
and
some
of
you
might
have
seen
the
slide
before
and
being
unkind
of,
you
know,
think
I'm
being
kind
of
lazy
too.
Updating
this,
but
I'm
not
there
actually
haven't
been
that
many
attacks
in
the
wild
being
reported
on
we've
had
lots
of
vulnerabilities
in
this
space
in
kubernetes
and
CPUs
and
related.
You
know
projects
and
whatnot,
but
no
significant
report
of
these
being
exploited
live.
B
An
interesting
thing
to
note
about
these
attacks
is
that
none
of
them
explicitly
attack,
containers
or
container
eyes
workloads,
and
that's
because
the
end
goal
isn't
your
application
right?
An
attacker
is
not
gonna
make
money
by
going
and
exploiting
applications
running
on
kubernetes
they're
gonna
make
money
by
stealing
your
compute
cryptocurrency
mining,
creating
a
botnet
whatever
it
happens
to
be
so,
let's
jump
into
some
forensics.
A
A
So
you've
probably
listened
to
all
of
Maya's
past
container
security
presentations,
you've
applied
security,
container,
kubernetes
security,
best
practices
to
your
cluster.
Your
cluster
is
about
as
secure
as
it
can
get
yet
something
or
somebody
still
infiltrated
your
environment.
Now.
What
well?
Don't
panic?
Don't
terminate
all
your
nodes
containers
disks?
This
is
where
some
of
the
evidence
lie
resides
that
we
need
to
collect
to
tell
the
story
of
what
happened
now
in
extreme
cases.
A
This
might
be
necessary,
but
Maya
will
talk
about
this
in
a
later
section
and
don't
be
a
hero,
don't
log
into
your
server
or
container
there's
multiple
reasons.
You
don't
want
to
do
this
one.
You
don't
want
to.
Let
the
attacker
know
that
you
know
what's
going
on
and
two
it
could
potentially
compromised
your
credentials.
A
Okay,
so
you're
not
panicking.
But
what
are
you
going
to
look
at?
Let's
start
to
gather
some
evidence.
But
what
are
you
going
to
look
at
this?
Is
you
know?
We've
we
find
that
the
evidence
lies
in
three
different
areas:
your
one,
your
logs
to
your
disks
and
three
live
and
potentially
historical
information.
So
we're
gonna
spend
some
time
focusing
on
this
area
and
again
how
it
pertains
to
containers
and
kubernetes
so
logs
I
doubt
anyone
would
really
question
the
value
of
logs
been
tracking
down
an
event.
A
So
what
we
want
to
do
here
is
identify
several
sources
of
the
meaningful
logs
and
decide
on
what
to
do
with
them.
You
want
to
make
sure
that
all
your
logs
are
going
to
a
central
location,
doing
this
you're
able
to
search
for
events
across
multiple
sources.
It
helps
guarantee
that
your
logs
haven't
been
tampered
with.
A
Even
if
the
hosts
are
coming
from
has
been,
and
then
depending
on
where
you're
sending
your
logs
some
of
the
places
you
can
send
your
logs,
you
can
do
some
can
do
anomaly
detection,
which
can
help
notify
you
of
a
possible
event
in
the
next
section.
Maya
will
talk
more
about
the
different
types
logs
and
the
information
you
might
gather
from
these.
A
So
traditionally,
in
a
data
center.
If,
after
event,
you
would
have
to
shut
down
the
server,
remove
the
disk
bring
that
back
up
in
some
point
in
a
quarantine
environment.
Luckily,
with
cloud
we
have
api's
where
we
can
take
dis,
nap
shots
container.
There
is
no
snapshot
mechanism,
but
we'll
talk
about
some
possible
solutions
in
this
area.
The
main
point
here
is,
you
can
start
doing
analysis
without
the
attacker
being
clued
in
and
what
it,
and
so
they
don't
erase
their
tracks
cool.
So
this
is
really
where
the
magic
comes
in.
A
A
The
SRE
team
at
Google.
Has
this
motto:
that's
we
hope
for
the
best,
but
plan
for
the
worst.
I
really
think
this
applies
to
security
and
more
specifically,
incident
response
plans.
It's
it's
really
easy
for
me
to
stand
up
here
and
tell
you
not
to
panic
when
an
incident
happens,
but
any
of
us
that
have
ever
been
involved
of
an
incident,
you're,
gonna
panic.
A
You
feel
that
in
the
gut
of
your
stomach,
you
just
you
know,
but
having
an
incident
response
plan
in
place,
helps
you
know,
or
the
other
folks
know
what
steps
to
take
in
case
of
incident.
It
lets.
You
know
who
to
notify
your
communication
plan,
the
systems
that
you
need
to
keep
up
and
running,
so
your
business
is
still
up
and
running
and
then
what
actions
to
take
and
how
to
start
diagnosing
what
happened
so
cool
so
I
mean
that
was
a
quick
overview
of
security,
forensics
and
really
focusing
on
collections.
B
Good,
so,
let's
look
at
logs
first,
as
Ann
mentioned,
that
one
of
the
first
places
to
look
when
you're
conducting
forensics
are
logs
many
logs,
don't
significantly
change
for
containerize
environments,
for
example,
like
never
logs,
aren't
really
different
as
long
as
you
can
map
back
an
IP
address
to
a
single
pod,
rather
than
just
the
node
that
it's
running
on,
so
we're
gonna,
look
at
four
different
kinds
of
logs
infrastructure
logs.
These
are
changes
to
your
infrastructure,
it's
what
the
infra
does
and
what
a
human
does
to
the
infra.
B
If
you're
using
a
cloud
provider
this
ink,
it's
changes
to
cloudy
ap
is
that
you're
accessing
and
to
your
services
kubernetes
logs.
These
are
what
the
control
plane
does,
what
a
container
does
to
the
control
plane
and
what
a
user
does
to
the
control
plane.
Think
of
this
is
anything
that
needs
coop
control
or
anything
that
kubernetes
handles
for
you
operating
system
logs.
These
are
calls
to
the
OS
on
the
node
that
the
container
runs
on
it's
about
what
actions
your
containers
are
taking
in
the
node
and
application
logs.
B
These
are
logs
for
what's
happening
inside
of
your
application.
They're.
Actually,
generally
the
same
for
containerized
and
non
containerized
apps,
but
how
you
collect
them
is
gonna,
be
a
little
bit
different.
All
of
these
also
have
the
concept
of
audit
logs.
So
if
you're
looking
to
figure
out
why
something
is
the
way
that
it
is
in
your
environment
and
who
made
it
that
way.
Poof
audit
logs
is
the
place
to
go
it
it's
or
what
you're
gonna
mostly
use
to
figure
out
what
another
human
like
an
attacker
did
in
your
environment.
B
So
let's
look
at
each
of
these,
so
first
infrastructure
logs,
let's
say
you're
running
in
a
cloud
infrastructure.
These
logs
should
include
information
about
what
underlying
services
you're
using
and
what
operations
the
user
made
to
those
resources
typical
logs.
That
you'd
expect
to
see
here
are
things
like
error,
logs
quota
warnings,
debug
information
as
well
as
audit
logs.
So
this
here
is
a
sample
cloud
audit
log
for
creating
a
cluster
in
gke.
You'll
notice.
Lis
contain
some
information
specifically
about
the
request,
like
the
type
of
operation,
how
it
was
authenticated.
B
Author
author
eyes,
as
well
as
information
about
the
content
of
the
request,
like
the
version
type
chosen
for
the
gke
cluster,
what's
gonna
be
logged,
it's
gonna
be
based
on
your
cloud
audit
logging
policy
or
logging
policy.
As
long
as
can
get
expensive
quickly.
You
probably
want
some
minimum
set
of
operations
logged
everywhere
in
your
environment,
but
only
much
more
detailed
logs
for
higher
risk
areas.
Example
of
high
risk
areas
where
you
want
might
want
more
logs
are
gonna,
be
your
PCI
compliant
cluster,
your
secret
storage
bucket
and
your
firewall
rules
next
up,
kubernetes
logs.
B
If
your
organization
has
a
team
that
manages
the
infrastructure
for
you,
including
kubernetes,
then
you
might
not
even
have
realized
that
these
were
different.
If,
however,
you
are
the
infrastructure
team,
though
then
criminai
these
logs
is
the
exact
same
concept
as
the
audit
of
the
infrastructure
logs,
but
for
kubernetes.
After
all,
kubernetes
is
part
of
your
infrastructure
right,
so
kubernetes
audit
logs
are
actions
taken
in
kubernetes.
A
given
request
will
generate
events
at
multiple
points
in
its
lifecycle.
B
First
to
when
the
request
is
received,
then,
when
a
response
has
started
and
finally,
when
a
response
is
completed,
the
cube,
API
server
will
process
these
events
and,
according
to
your
audit,
logging
policy
will
log
them
so
just
like
for
your
infrastructure,
logs
kubernetes
can
log
events
at
different
depths
of
detail
based
on
the
information.
That's
included.
Ranae's
has
four
different
audit
levels
that
are
on
the
left-hand
side
of
the
page
here,
none
which
doesn't
log
events
at
all
metadata,
which
logs
request
metadata
only
this
includes
requesting
user
timestamp
a
resource
and
verb.
B
This
is
a
good
choice
for
operations
where
the
verb
is
enough
to
tell
you
what
happened.
The
next
level
request,
which
logs
metadata
plus
the
request
body.
This
is
a
good
choice
for
operations
where
you're
changing
configurations
and
you
want
to
be
able
to
capture
what
that
changes
and
then
request
response
which
logs
event
metadata
and
the
request
body,
plus
the
request
response
body.
If
you're
getting
started
crafting
your
own
audit
policy,
you
can
check
out
the
one
that
GK
uses
online.
There's
some
excerpts
here.
B
It's
in
the
in
the
open
source,
repo
basically
get
requests
because
they
can
be
very
large
to
log
or
at
the
request
level
known
api's
are
at
the
request
response
level
and
any
calls
not
explicitly
covered
by
the
policy
or
at
the
metadata
level,
again
meeting
that
kind
of
base
level
of
logging
that
you'd
expect
to
have
everywhere
operating
system
logs
cover
events
happening
inside
your
node
operating
system.
Right
when
you
remember,
containers
have
two
sets
of
OSS.
B
This
is
the
one
on
the
node
that
you're
running
your
containers
on
top
of
so
this,
for
example,
will
include
network
connections,
user,
logins
and
SSH
sessions
and
certain
executions
like
exec
ve.
When
you
collect
these
logs
on
kubernetes,
you
do
so
by
deploying
a
logging
agent
as
a
daemon
set,
for
example,
flow
and
D
to
collect
the
logs,
which
then
writes
them
to
a
logging
infrastructure
like
stackdriver
or
elasticsearch
and
Cabana.
Fluid
e
has
really
good
documentation
on
how
to
do
this,
so
I'm
not
going
to
go
into
too
much
detail.
B
If
you
check
out
the
GK
documentation
you
can,
you
can
take
a
look
at
that,
be
careful
if
you're
gonna
tune
any
of
these
os
Auto
logs
yourself,
because
you
could
end
up
with
a
lot
of
logs
and
I
mentioned
earlier.
Logs
are
expensive
and
storing
logs
are
expensive,
so
make
sure
you
actually
know
what
you're,
what
you're
capturing
or
test
it
out.
First
before
you
kind
of
dive
dive.
All
in
note
also
that
logs
can
only
be
generated
for
events
occurring
after
the
demon
set
is
deployed
when
you're
using
a
model
like
this
right.
B
So
it's
recommended
to
create
the
demon
set
right
after
cluster
creation.
Then,
every
time
that
you
add
new
nodes
to
the
cluster
they'll
have
the
configuration
apply
automatically
and
then
lastly,
application
logs
application
logs
are
logs
from
your
application
that
you
wrote,
including
errors
warnings,
for
example,
of
like
low
memory
or
disk
space.
They
actually
aren't
that
different
for
containerized
applications
as
I
mentioned
earlier,
but
how
you
collect
them
is
gonna,
be
a
little
bit
different.
Actually
how
you
collect
them
is
really
easy
because
we
just
talked
about
it.
B
B
Where
do
the
logs
come
from
and
where
do
they
go
in
your
infra
infrastructure
locks,
including
audit
logs,
come
from
your
infrastructure,
surprise
and
somehow
end
up
in
your
logging
system,
kubernetes
logs
and
audit
logs,
that
you
collect
from
kubernetes,
for
example,
using
a
logging
agent
like
fluent
e,
if
you're
using
a
managed,
kubernetes
service.
This
is
usually
ingested
for
you,
OS
logs,
including
audit
logs
and
application
logs.
B
You
can
collect
yourself
using
your
own
logging
agent,
now
recall
if
you're
running
a
logging
agent
yourself,
you're
gonna,
probably
want
to
send
them
to
a
logging
infrastructure
like
I
mentioned,
and
what
Ann
said
at
the
beginning.
I
talked.
Is
you
probably
want
to
get
all
of
your
logs
in
one
single
place?
That
makes
it
easier
for
you
to
filter
over
all
the
possible
events
that
occurred
in
infrastructure
and
also
don't
be
able
to?
You
know
pivot
on
any
particular
resource
and
see
and
see
what
actually
happened
to
that
thing.
B
A
Cool,
so
this
is
an
example.
The
in
this
example
we're
using
API
to
create
distaff
shot
and
then
of
the
affected
node
and
sending
a
copy
over
to
an
isolated
location.
So
this
isolated
location,
if
you're
an
AWS,
could
be
a
different
AWS
account
TCP,
maybe
a
different
GCP
project,
or
it
could
just
be
an
isolated
Network.
You
know
lockdown
VPC
and
then
you
can
start
doing
analysis
of
it.
But
the
thing
here
is
once
you
bring
this
disk
image
up
on
a
different
node.
A
Your
containers
are
no
longer
running,
so
it's
really
hard
to
tell
what
was
going
on
at
the
time
the
snapshot
was
taken
at
you
know
when
the
containers
were
running
so
at
Google
we
have
a
open
source
project
called
docker
Explorer,
which
is
helps
you
do
forensic
analysis
of
offline
docker
file
systems
from
a
distaff
shot.
So,
for
example,
if
you
take
this
snapshot
of
a
GCE
image,
you
get
a
copy
of
our
lib
docker.
So
docker
Explorer
is
just
Python
that
does
some
JSON
parsing
so
running.
A
The
Dhokla
Explorer
on
the
mounted
VAR,
Lib,
docker
and
doing
list
running
containers
will
get
container
IDs
that
were
running
at
that
point
in
time.
The
snapshot
was
taken
that
you're
then
able
to
mount
those
file
systems
and
see
what
was
going
on
in
the
container
again.
At
that
point
in
time
when
the
snapshot
was
taken,
so
isolating
one
node
and
container
is
easy.
But
again
you
know,
if
you
have
hundreds
of
those
thousands
of
containers.
How
do
you
start
analyzing?
A
What's
going
on
in
those
quickly
again
without
having
to
log
in
what
do
you
do
if
your
container
has
died?
How
do
you
get
that
information?
There's
a
lot
of
security,
container
security,
vendors
that
have
applications
in
this
area
and
they're
here
I
would
encourage
you
to
speak
to
them,
but
for
this
presentation,
I
really
want
to
focus
on
some
open
source
projects.
So
the
first
one
is
called
Gurr.
It's
an
open
source
project
from
Google.
A
It
stands
for
gue,
Rapid
Response,
it's
an
Incident
Response
framework
for
VM,
so
the
TLDR
on
this
slide
is
secure
it
before
you
unleash
it
so
again
with
Girt.
You
can
remotely
get
detailed
monitoring
of
your
client
or
node
CPUs
memory,
I/o
usage
and
memory
dumps,
so
Gura
has
two
main
activities,
flows
and
hunts.
So
a
flow
is
when
you've
asked
ger
to
go,
find
out
something
on
one
of
your
nodes.
So
maybe
you
want
the
memory
on
a
running
process.
Maybe
you
want
information
about
a
certain
file
like
the
contents
of
a
file.
A
That's
when
you
would
do
a
flow.
A
hunt
is
always
running.
It's
looking
for
something
bad.
So
maybe
it's
looking
for
a
key
file
that
changed.
Maybe
it's
looking
for
a
malware
signature
or
something
like
a
jar
file
and
again
ger.
It's
really
focused
on
VM.
So
you
know,
one
of
the
things
that
doesn't
do
is
map
docker
processes
to
OS
pins.
So
this
is
where
some
assisted
open-source
projects
come
in.
A
So
cystic
cube,
see
tail
capture
record
sis
calls
in
a
capture
file
which
can
help
you
diagnose
what
was
going
on
during
your
during
an
attack.
Even
if
the
container
has
died
and
then
Cystic
inspect
is
an
interface
that
displays
the
contents
of
a
capture
file.
So
the
content
is
organized
and
these
different
tiles
each
would
show
a
value
in
relevant
metrics
and
its
trends.
A
The
tiles
are
organized
in
different
categories:
to
help
surface
information
more
clearly,
you're
able
to
then
drill
down
into
these
different
tiles
and
get
the
data
behind
it
and
really
start
investigating
what
was
going
on,
you're,
also
able
to
narrow
your
search
based
on
a
time
window
and
then
similar
to
gir.
But
you
know
with
four
containers:
you
can
get
information
on
your
CPU
CPU
memory,
I/o
system
calls
and
even
the
contents
of
files.
B
Cool
so
now
that
you've
detected
something
you
know
using
your
logs
using
agur
using
snapshots,
whatever
happens
to
be
what
should
you
actually
do
in
your
environment?
So,
let's
go
through
the
most
common
mitigation
options
that
you
have
at
your
disposal.
You
could
send
an
alert,
isolate
a
container,
posit
container,
restart
a
container
or
kill
a
container.
These
options
are
increasing
an
order
of
certainty,
but
also
an
order
of
severity
right
killing
a
container
is
your
last
possible
mitigation
and
even
for
the
most
severe
of
incidents
might
not
actually
be
the
best
option.
B
As
an
explained,
there
isn't
a
great
way
of
snapshot
in
containers,
so
live
forensics
is
often
your,
but
if
you
kill
the
container,
you've
lost
that
opportunity
for
analysis.
So
for
each
of
these
options,
we're
gonna
quickly
go
over
what
it
is.
When
you
do
that
option
and
how
you
might
do
it
so.
First
alerting
alerting
notifies
your
Security
Response
Team
that
something
is
going
on
in
your
infrastructure.
It's
it's
about
sending
potential
events
to
your
security
team
for
triage
and
further
analysis.
B
This
is
typically
the
first
step
before
your
team
takes
one
of
the
more
severe
actions.
If
you're
getting
all
of
your
logs
in
one
place
like
you
should
be,
then
you're
alerting
should
just
leverage
that
use
the
same,
alerting
tools
that
your
ops
team
is
already
using,
whether
that's
you
know,
feeding
alerts
to
email,
JIRA,
slack
pager
duty,
whatever
works
for
you
alerts,
should
be
based
on
specific
metrics
or
on
specific
actions.
B
So,
for
example,
CPU
usage
in
your
container
or
a
new
network
connection,
the
hard
part
here
is
actually
figuring
out
what
you
want
to
alert
on.
It's
not
you
know
commonly
agreed-upon
what
those
metrics
should
be.
It's
gonna
end
up
being
very
specific
on
your
particular
info.
The
next
option
is
isolation.
Now
I
don't
mean
isolation
in
the
sense
of
workload.
Isolation
like
running
untrusted
code
in
a
multi-tenant
cluster
I
mean
more
like
quarantine.
B
You
want
to
move
the
container
to
somewhere
that
you
can
examine
it
better
without
it
necessarily
knowing
that
anything
is
up
that
you're
actually
looking
at
it.
You
probably
want
to
do
this
when
your
container
is
doing
something
odd,
but
the
workload
that's
affected
isn't
necessarily
critical
to
your
business,
so
this
should
help
limit
damage
in
your
environment,
for
example,
by
preventing
an
attacker
from
moving
laterally
and
getting
access
to
further
systems
or
by
preventing
one
application
from
dossing
another
application.
This
is
actually
pretty
hard
to
do.
B
Well,
basically,
what
you
need
to
do
is
move
your
container
to
a
new
network,
so
the
easiest
way
to
do
this
is
actually
to
reschedule
the
workload
on
a
new
node.
That's
a
preferred
option
than
keeping
the
workload
on
the
existing
node
and
then
removing
all
the
other
workloads,
the
reason
being
that
in
case,
those
workloads
are
also
affected,
you're
losing
potentially
important
information.
B
So
when
you
reschedule
your
workload
to
make
sure
that
nothing
else
is
ends
up
on
the
same
know,
to
use
queue,
control,
coordinate
and
then
you
can
create
a
restricted,
Network
policy
to
limit
the
containers
connectivity
and
then
you
can
set
up
other
tools
that
you
want
for
investigation
like
like
live
forensics
on
node
agents
or
network
filter,
ranked
and
actually
figure
out.
What's
going
on
the
next
step,
the
next
option
is
pausing
a
container
pausing,
a
container
suspends
running
processes
in
the
container.
B
This
is
a
really
good
choice
when
you
have
a
decent
idea
of
what
the
issue
is,
and
you
just
need
to
get
some
a
quick
data
for
record-keeping
or
to
confirm
what
it
is
or
determine
what
action
to
take
next.
This
is
particularly
useful
for
connecting
further
forensics
one
good
example.
Here
is
cryptocurrency
mining.
B
It's
not
necessarily
harmful
to
your
workload,
but
it
could
be
costing
you
quite
a
bit
of
money
so
by
pausing
your
container
you're
not
incurring
additional
costs,
but
you
also
have
the
opportunity
to
take
a
backup
and
figure
out
the
source
of
the
issue
before
you
actually
restart
the
container
pausing,
a
container
is
actually
pretty
easy
because
it's
built-in
in
docker,
you
can
use
docker
pause
to
pause
containers
using
the
cgroups
freezer
with
the
cgroups
freezer.
The
process
is
unaware
and
unable
to
capture
that
it's
being
suspended
and
then
subsequently
resumed.
B
This
is
exactly
doing
what
it
says.
It's
on
the
box,
which
is
great,
restarting
a
container,
restoring
a
container,
kills
and
restarts
a
running
container
and
resets.
The
current
state
of
the
application,
restarting
a
container,
makes
sense
when
you
already
have
a
fix
that
you're
working
on
rolling
out.
So
you
have
to
be
really
careful
with
restarts.
They
don't
make
sense
when
you
sorry,
they
don't
make
sense.
B
Unless
you
already
know
where
the
issue
is
coming
from
so
say,
an
attacker
has
compromised
your
container
and
if
you
restart
it,
you're
kicking
them
off
of
that
foothold
again.
Unless
they've
also
compromised
your
note
or
something
like
that,
but
you're
not
actually
fixing
the
problem
because
they
can
get
right
back
onto
your
container
the
same
way
they
got
there
in
the
first
place.
B
In
fact,
you
might
have
just
made
it
worse,
because
now
you
told
them
that
you
know
that
they're
there
right
and
you
probably
have
multiple
instances
of
the
same
container
running
in
multiple
places
they
can
get
into
all
of
those
as
well,
so
you're
not
actually
fixing
the
problem.
You're
just
kind
of
temporarily
telling
them
come
at
me
harder
like
it's.
Not
it's
not
gonna
help
you,
so
you
can
restart
a
container
with
docker
restarts
using
this
command.
B
You
can
also
specify
the
seconds
to
wait
for
processes
to
stop
before
the
container
is
actually
killed
and
restarted
in
kubernetes.
You
can
also
delete
the
pod
and
it'll
just
get
rescheduled.
I
want
to
repeat
this,
though
you're
supposed
to
do
this
when
you
already
have
a
fix.
So
if
you
already
have
a
fix,
the
easiest
way
to
do
this
is
to
roll
out
your
fix.
You
don't
actually
have
to
kill
the
container
and
do
all
this
other
stuff,
just
roll
out
a
new
fix,
kill
the
old
work
load
and
magic.
B
Things
of
your
attacker
is
no
longer
in
that
workload.
Lastly,
killing
a
container.
This
is
the
last
resort
that
you
have,
as
it
halts
all
running
processes
in
the
container.
Like
a
restart.
This
will
let
the
attacker
know
that
you
know
and
doesn't
fix
the
underlying
problem,
plus
it
takes
your
workload
offline.
Now
you
might
do
this
in
a
worst
case
scenario.
B
For
example,
you
an
application,
that's
actively
exfiltrating
sensitive
user
data
in
this
case,
stopping
the
leak
might
actually
be
more
important
to
you
in
your
organization
than
figuring
out
how
it
happened
or
getting
forensics
again
very
specific
on
your
situation
to
kill
a
container,
you
can
use
docker,
stop
or
cry
control,
stop
and
docker
kill.
Dockers.
Stop
stops
the
container
by
sending
a
signal
to
the
root
process.
K--
asks
nicely
and
gives
it
ten
seconds
to
stop
before
sending
ass,
and
if
the
process
hasn't
exited
in
that
time
period,
it
then
sends
a
signal
signal.
B
Docker
kill
is
less
graceful;
it
doesn't
wait
around.
It
doesn't
mess
around.
It
just
sends
the
signal
signal
directly.
If
you're
at
this
point,
where
you're
killing
a
container,
you
probably
want
docker,
kill
you're,
not
gonna,
give
it
an
extra
ten
seconds
to
wait
around.
You
can
also
remove
a
container
forcefully
with
docker
RM
dash,
F
or
cry
control.
Rmf,
though
docker
remove,
is
an
for
an
already
stopped
container
using
the
dash
F
flag
forces
it
to
terminate
using
a
cig,
kill,
SiC
signal.
First.
A
Awesome,
alright,
let's
apply
all
this
knowledge
to
a
real-world
scenario.
So
let's
look
at
a
common
container
attack,
privilege,
escalation
or
container
escape
so
privilege.
Escalation
attacks
allows
containers
to
break
out
of
their
isolated
contexts
and
essentially
become
route
on
the
server
and
effectively
having
access
to
all
the
other
containers
running
on
that
node.
A
So
maybe
you've
been
contacted
by
your
cloud
provider
and
been
notified
that
some
they've
noticed
some
suspicious
network
traffic
coming
from
when
your
nodes
or
multiple
nodes,
or
maybe
you've,
seen
a
spike
in
CPU
usage
on
your
monitoring
system.
Neither
one
of
these
events
actually
indicates
a
security
incident,
let
alone
privilege
escalation,
but
what
they
do,
give
you
some
information.
They
give
you
the
nodes
or
set
of
nodes
that
might
the
event
might
be
happening
on
in
a
time
range
date
and
time.
So
what
do
you
have
in
place?
A
Well,
since
you've
listened
to
our
presentation,
you
know
that
you
owe
your
logs
should
be
going
to
a
central
location.
You
know
to
isolate
and
quarantine
you're
just
snapshot,
and
then
you
have
invested
some
time
and
container
security
specific
tools.
So,
let's
first
look
at
logs.
So,
since
you
have
that
time
range
and
hoes,
you
know
start
focusing
on
the
logs
there,
maybe
you
see
something
suspicious
there
and
you
can
take
that
pattern
and
search
across
other
logs
and
to
see
if
ever
other
nodes
or
hosts
had
been
affected.
A
So
maybe
maybe
you've
narrowed
this
down
to
a
set
of
nodes
or
a
nose.
So
let's
start
looking
at
the
OS
logs
or
deployment
logs
for
that
particular
node.
So
you
want
to
maybe
start
looking
at
where
your
containers
launched
in
such
a
way
that
they
shouldn't
be
maybe
they're
being
ran
as
root
or
privilege
or
mounting
filesystem
cessation.
These
could
indicate
privilege
escalation,
so
maybe
you've
you've
narrowed
it
down.
Now,
at
this
point
to
a
specific
container,
that's
still
running
so
you
should
have
a
pretty
good
idea
of
what
your
container
image
is.
A
The
processes
that
are
running
in
those
so
looking
at
the
container
logs
you
know,
are
there
some
suspicious
processes
that
are
running
you
know,
maybe
the
W
get
or
a
curl,
that's
downloading
malware,
maybe
something's
been
on
tart
or
some
sign.
Shell
script.
That's
running
note
those
file
names
because,
using
your
other,
you
know
something
like
girl
or
the
Cystic
products.
You
can
get
the
contents
of
those
files
and
you
know
with
Network
log
you
know:
are
you
seeing
increased
unexpected
outbound
traffic
from
your
nodes?
A
This
could
indicate
that
your
data
is
being
sent
out
places
it
shouldn't
be
so
you
have
a
point
in
time
snapshot
of
what
was
going
on,
potentially
during
an
attack
in
the
case
of
privilege,
escalation.
The
malware
within
the
can
get
access
to
the
underlying
OS,
possibly
changing
files
on
the
OS.
So
you
have
this
distaff
shot.
A
But
what
you
also
have
is
the
VM
image
that
you
create
your
nodes
from,
so
you
could
effectively
do
a
diff
of
these
file
systems
and
look
for
binaries
that
have
changed
so
things
in,
like
user
bin
that
shouldn't
have
changed
or
in
the
case
of
the
run,
C
vulnerability
that
was
announced
in
February.
The
malware
might
have
changed
a
run,
C
binary,
so
you
would
be
able
to
detect
that
and
if
that's
the
case
more
likely,
the
privilege
escalation
has
happened
so
similar
to
what
you
might
find
in
your
container
logs.
A
You
could
find
in
the
capture
files,
but
with
more
information
and
correlation
of
things
that
are
going
on.
So
you
can
find
information
about
what
user
was
running.
The
process
files
the
process
was
touching.
Io
unusual
system
calls
and
then
again
the
contents
of
a
file
and
then
again
you're
able
to
correlate
events
that
you
might
not
be
able
to
correlate
with
your
log
files
and
again
in
the
case
of
the
Runcie
binary
getting
overwritten,
you
might
actually
be
able
to
detect
the
event
that
caused
that
to
happen.
A
So
going
back
to
the
mitigation
options
at
miles,
Maya
talked
about
in
the
last
section,
the
ones
that
we
want
to
focus
on
for
privilege.
Escalation
are
alert
and
isolate.
So
in
the
case
privilege
escalation,
you
could
possibly
get
alerted
again.
You
know
if
your
containers
are
being
run
as
root
or
just
privilege.
You
can
also
get
alerted
on
specific
CVEs.
A
A
If
there
really
was
a
container
escaped,
you
may
need
to
remove
all
the
nodes
that
your
container
had
touched,
which
possibly
means
creating
a
new
node
pool
in
migraine
in
to
clean
pods
over
and
then
once
you've
identified
a
fix
for
that
container
image.
You'll
want
to
roll
out
a
fix
or
update
to
the
new
node
pool,
and
then
just
last
week,
Cystic
released
a
blog
post
on
how
you
can
automate,
alerting,
isolating
with
Falco
GCP,
pub/sub
and
cloud
functions
won't
have
a
link
to
that
the
end
of
this
presentation.
A
So
what
what
can
you
do
today
so
first
create
that
incident
response
plan
I
mean
again,
you
know
when
an
incident
happens,
you
need
to
know
the
steps
to
take
so
we're
not
all
in
panic
mode.
Secondly,
follow
container
security,
best
practices
when
you're,
creating
your
clusters,
have
all
your
logs
syncing
to
a
central
location
and
then
vesting
container
security,
specific
tools,
open
source
or
not
and
again,
I
encourage
you
to
speak
to
some
of
the
vendors
that
are
here
and
then
rehearse
the
process
with
a
fake
event.
A
You
don't
want
to
be
the
first
time
you
run
through
this.
When
an
incident
happens,
you
want
to
rehearse
this,
so
you
can
find
where
all
the
pointy
edges
are
fix
them,
and
you
know,
don't
panic.
You've
got
this,
so
here
are
some
resources
of
some
of
the
things
that
we
mentioned
throughout
the
presentation.
C
A
D
Yeah
great
presentation,
I'm
I'm,
strictly
relating
at
the
third
mitigation
option.
The
polls
is
there
such
an
option
in
communities
to
pause
a
pod
or
to
take
it
all
from
the
service,
because
I
can
do
it
with
docker,
but
I
want
I
want
to
introspect
it
I
want
to
look
at
it,
but
I
don't
want
my
pot
to
keep
incoming
having
incoming
requests
yeah.
B
B
There's
a
couple
like
open
source
references
and
things
like
that,
so
Falco
has
a
bunch
of
open
source
rules
for
4gk,
specifically
on
GCP.
We
also
announced
a
product
tip
made
a
few
months
ago
called
event
threat,
detection,
which
looks
specifically
at
logs
for
things
that
look
bad.
It's
not
container
specific,
it's
for
general
issues,
but
there's
a
bunch
of
also
partner
solutions
in
this
space.
I
think,
if
you're,
asking,
though
like,
is
there
a
generic
enough
set
of
like
rules
and
open
source
for
things
that
look
bad
in
kubernetes
clusters?
C
B
You
manage
to
compromise
a
node
and
you
manage
to
compromise
the
demon
set
on
that
node.
You
can't
really
trust
anything
that
it's
writing
so
the
way
that
I
typically
see
security
people
do
this
is
I
mean
assuming
the
demon
set
will
be.
Okay,
well,
I'm,
really
trying
hard
to
two
things
that
can
become
here:
two
things
that
can
be
compromised.
You
can
either
compromise
that
demon
set
that's
running
on
the
node
itself.
B
In
that
case,
you've
compromised
the
node
and
you
have
bigger
problems
than
just
the
container
being
compromised,
because
now
your
attacker
has
the
no
level
either
they
escape
the
container,
which
is
highly
unlikely
in
a
while
in
the
wild.
But
you
know
something
that
is
worth
investigating,
or
they
have
some
other
type
of
access
on
your
node
and
that
therefore
there
another
parts
of
your
infra,
the
second
thing
they
can
try
to
compromise,
is
where
your
logs
get
written
to.
B
So
that's
so
remember
we
have
a
logging
agent,
and
then
we
gets
written
to
logging
infrastructure,
so
you
know
protect
the
logging
agent
by
it's.
Typically
a
daemon
set
again
at
the
nuit
level,
the
logging
infrastructure.
What
I
typically
see
people
do
is
actually
write
the
logs
to
a
separate
project,
the
idea
being.
If
an
attacker
has
gotten
into
a
single
project,
they
can
probably
access
those
logs,
even
if
they
can't
modify
them,
they
can
at
least
see
them,
which
tells
them
that
they
know
that
you
know
so.
F
B
Can't
really
talk
about
incidents,
we
do
a
lot
of
vulnerability
response,
I
think
it's
actually
worth
respond.
Oh
dang
and
also
vulnerability
response
and
incident
response
are
different
things
right
until
the
vulnerability
is
exploited.
You
probably
want
a
different
team
working
on
that
than
the
incident
response,
because
vulnerability
of
on
to
something
you
can
do
you
can
plan
for
or
as
incident
response
is
very
reactive
in
terms
of
instant
response.
The
what
I'll
just
basically
say
is
the
stuff
that
we
see
is
what
you
would
expect
us
to
see
like.