►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Lightning Talk: Challenges in Cloud Native Forensics - Andrew Krug, Datadog
As more companies have gone cloud native the focus on resilience has largely focused on detection and speedy recovery. These are only two tactics that should be in the defense toolbox. Forensics is a discipline that has arguably suffered as log volumes and DevOps culture has become more normative. In my session, I’ll demonstrate where the gaps exist and how the ecosystem could improve capabilities around the art of forensics.
A
Welcome
to
challenges
in
cloud
native
forensics,
here
at
cloud
native
security
day,
happy
to
be
here,
I'm
andrew
krug,
I'm
a
security
geek
at
datadog
and
also
a
technical
evangelist
and
up
on
screen
here
is,
of
course,
where
you
can
reach
me.
If
you
have
questions
or
feedback
about
the
talk,
I
started
my
career
in
forensics
a
long
time
ago.
A
Things
such
as
breaches
due
to
misconfigurations,
which
we
see
all
the
time
or
even
incidents
like
employment
law
related
things
will
force
you
to
sift
through
logs
and
disk
images
and
memory
samples
from
your
environment.
Regardless
of
the
reason,
though,
I
tend
to
think
of
forensics
as
kind
of
the
definition
in
that
purple
box
there,
which
is
telling
stories
that
occurred
in
a
specific
time
window
using
facts
that
can
be
derived
provably
and
repeatedly.
A
We
used
to
talk
about
this
as
using
things
that
we
called
validated
tooling
and
also
keeping
track
of
the
chain
of
custody
for
pieces
of
evidence
and
tool.
Validation
put
simply
is
just
that
we
can
use
a
tool
over
and
over
and
that
tool
has
been
studied
and
proven
to
yield
the
same
results,
hundreds
or
thousands
of
times
in
in
academic
tests
in
stress
tests
and
also
proven
not
to
modify
the
actual
artifacts.
A
So
that
we
can
go
back
and
we
can
see
the
complete
log
of
just
what
happened
over
the
course
of
an
investigation.
This
is
one
of
the
things.
That's
notably
gotten
a
tiny
bit
easier
with
cloud
provider
control
plane
logs,
like
aws
cloudtrail,
which
brings
us
to
kind
of
the
problem
statement
here,
which
is
that
the
more
we
embrace
kind
of
this
idea
of
devops
and
and
cattle
not
pets
the
more
challenged.
A
Some
of
these
forensic
processes
do
become,
and
that's
due
to
like
kind
of
three
different,
distinct
pillars
that
continue
to
challenge
us.
When
we
talk
about
cloud
native,
forensics,
ephemerality
or
short-lived
instances,
scale,
which
is,
of
course,
the
the
number
of
workloads
that
we
might
have
to
perform,
forensics
on
and
also
the
scope.
Maybe
it's
a
single
aws
account
or
one
kubernetes
cluster,
or
it
could
be
hundreds
of
kubernetes
clusters
depending
and
also
technology.
The
the
very
technologies
that
we
put
in
place
to
help
us
do.
A
A
Today
we
have
much
shorter
and
shorter
lived
workloads,
and
what
you're
seeing
on
screen
is
is
some
some
facts
from
a
datadog
report,
which
is
how
long
different
classes
of
workloads
exist.
So
in
serverless
compute
you
might
have
a
workload
that
lives
for
four
minutes
in
orchestrated
containers.
They
might
live
for
for
half
a
day
to
a
day
and
in
unorchestrated
containers
they
might
live.
You
know
for
like
something
like
four
to
six
days.
A
Still,
that's
not
very
long
compared
to
the
span
of
years
that
something
might
be
in
production
providing
compute
versus
today,
so
we're
cycling,
things
out
quite
a
bit
faster
and
that
actually
is
a
problem
for
forensics,
because
it
destroys
evidence
when,
when
you
do
a
deploy,
your
containers
go
away.
When
you
do
a
ploy,
your
deploy,
your
ec2
instances
go
away
when
your
auto
scale
group
expands
and
contracts
we're
actually
throwing
away
valuable
instance
data
and,
according
to
google's
state
of
devops
report,
the
average
company
deploys
626
times
a
year.
A
A
You
could
already
have
kind
of
gotten
rid
of
all
of
the
evidence
that
would
help
you
kind
of
break
the
case
and
figure
out.
You
know
exactly
how
somebody
got
in
what
they
got
out,
etc.
So
that's
kind
of
a
problem
and
scale
adds
another
dimension
to
this
as
well.
So
most
environments,
running
orchestrated
containers
or
any
sort
of
orchestration
on
top
of
aws
or
gcp
or
azure
are
running
a
lot
of
workloads,
and
this
is
just
a
statistic
from
one
single
report
so
take
it
for
what
it
is.
A
A
Back
in
the
good
old
days,
we
talked
about
two
different
types
of
forensics
right
and-
and
I
want
you
just
to
remember
these
two
distinctly
different
types
of
forensics.
One
was
what
we
called
cold
forensics
or
dead
system,
which
was
where
we'd
actually
take
hard
drives
and
take
them
out
of
systems
image
them
carve
the
the
actual
file
system
for
deleted
files
artifacts.
That
would
help
us
reconstruct
a
timeline.
A
The
second
is
live
forensics
and
that's
the
act
of
getting
things
from
volatile
system
data
that
would
go
away
when
you
turn
the
machine
off,
and
this
wasn't
just
memory,
there's
also
things
like
network
information
process,
information,
things
that
wouldn't
necessarily
be
resident
on
disk
and
for
a
time
we
thought
this
was
actually
going
to
be
this
massive
boon
to
the
industry.
Right,
because
we
could
do
this
so
much
faster,
using
live
memory,
samples
to
like
crack
the
case,
then
we
could
carve
through
a
disk.
A
It
turned
out,
though,
that
despite
the
efficacy
of
this
and
the
fact
that
we
could,
we
could
use
it
to
solve
problems.
We
couldn't
use
disk
forensics,
for
it
was
short-lived
because
operating
systems
started
to
add
more
and
more
security
features,
kind
of
beginning
in
2010,
with
data
execution
prevention
and
address
space
layout
randomization,
which
would
take
the
address
at
which
a
given
process
was
loaded
into
memory
and
sort
of
offset
that
randomly
at
any
given
time
and
then
kernel
level
address
space.
A
And
there
are
a
variety
of
good
tools
that
exist.
They
just
can't
necessarily
keep
up
with
all
of
the
innovations
going
on
with
kind
of
the
randomization
of
memory.
And
now,
if
you
look
at
hardware,
they're,
actually
putting
memory,
obfuscation,
features
and
hardware,
that
makes
it
increasingly
difficult
to
analyze
memory
samples
and
so
often
what
we
end
up
with.
Even
if
we
do
everything
right
is
a
complete
lack
of
ability
to
analyze
live
system
data
at
any
kind
of
practical
scale,
especially
as
memory
sizes
increase,
instant
sizes
increase,
etc.
A
So
we
really
need
to
make
the
art
of
forensics
a
first-class
citizen.
We
need
to
be
thinking
about
this
as
we
build
out
operating
systems
as
we
build
out
orchestrators
that
eventually,
during
a
security
incident,
someone
is
going
to
need
to
kind
of
lawfully
intercept.
If
you
will
the
data
that's
flowing
through
that
system
in
a
way
that
they
can
reconstruct
it,
and
that
needs
to
be
provable
and
validatable
just
like
we
talked
about.
A
So
if
you
think
this
is
a
good
idea,
please
go
and
plus
one
github
issues
on
any
of
those
projects
I
put
on
screen
that
have
to
do
with
aslr
or
kaslr
or
if
you
are
on
the
board
of
a
prominent
project
like
kubernetes
or
the
linux
kernel.
Let's
have
a
chat,
let's
think
about
forensics
a
little
bit
differently.