►
From YouTube: How This Innocent Image Had a Party in My Cluster - Amir Jerbi & Itay Shakury, Aqua Security
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
How This Innocent Image Had a Party in My Cluster - Amir Jerbi & Itay Shakury, Aqua Security
As security practices and tools for scanning container images are becoming increasingly popular, malicious actors are introducing sophisticated techniques to obfuscate their intent and evade scanning tools. The malware they plant cannot be detected using static analysis, or based on signatures. But dynamic analysis that runs the image, and then traces the activity of the running container can detect and document the entire multi-stage attack. In this talk, we will review the evolution of these attacks as observed by Aqua’s security research team, and demonstrate the full chain of events and IoCs (indicators if compromise) that were detected. We will give practical advice on what developers and cluster admins should do to detect similar techniques, and the security controls the should be employed to reduce the chances of such an attack succeeding, as well as reduce its potential impact.
https://sched.co/Zeks
B
The
focus
of
this
talk
is
going
to
be
scanning
of
your
container
images.
So,
let's
start
so
today
when,
when
we
are
talking
about
scanning
of
your
container
images,
we
are
basically
looking
for
three
things.
We
are
looking
for
vulnerabilities
for
malware
and
for
misconfiguration
inside
of
the
container
images.
B
So
what
does
it
mean?
Let's
give
you
a
small
background
of
that?
So
vulnerabilities
mean
bad
code,
usually
it's
components
that
are
outdated,
all
components
that
are
known
to
be
vulnerable.
Basically,
they
have
code
that
an
attacker
can
leverage
and
exploit
in
order
to
get
access
to
the
system
or
to
collect
data.
B
There
is
a
database
of
vulnerabilities
called
nvd.
Each
vulnerability
has
a
unique
identifier,
a
description,
a
severity
between
load
to
critical,
and
it
also
has
the
affected
component.
What
is
the
component
and
the
version
that
has
those
vulnerabilities
and
in
the
market
today?
We
use
vulnerability
scanners
in
order
to
identify
those
vulnerabilities
in
container
images.
B
So
there
are
open
source
vulnerabilities
like
3v.
There
are
also
commercial
options
for
vulnerability
scanners,
and,
what's
in
common
for
all
of
them,
is
that
they
will
take
your
container
image
open
it
up,
look
for
all
of
the
files,
inside
of
it
all
of
the
operating
system,
all
of
the
programming
languages
packages
and
basically
match
them
against
the
mvp
database
or
in
some
cases,
other
flat
intelligence
database
of
vulnerabilities
and
the
result
will
be
the
list
of
components
that
are
vulnerable.
B
B
The
second
thing
that
we
are
scanning
today
for
container
images
is
mis
configurations
so
that
there
are
scanners
today
in
the
market
that
can
find
whether
your
container
image
has
hard-coded
ssh
keys,
weak
ciphers
of
tls.
In
some
cases,
you
don't
have
any
configuration
of
tls
setup
in
your
container
image.
Hardcoded
passwords,
those
bad
configurations.
B
And
the
last
item
that
we
are
scanning
is
malware.
Basically,
malware
scanning
means
that
we
are
taking
the
container
image.
We
are
looking
at
all
of
the
files
inside
of
it.
We
are
taking
the
the
files
and
and
creating
a
sha
signature
out
of
them,
and
then
comparing
this
sha
against
the
database
of
malware
in
the
screenshot
you
you
can
see
a
very
popular
database
called
virustotal.
B
So
the
question
is:
if
I
have
a
scanning
for
vulnerabilities
scanning
for
misconfiguration
and
scanning
for
malware,
am
I
protected
and
the
answer
is
no,
there
is
still
a
blind
spot.
There
is
something
called
evasive,
malware,
evasive
malware
are
the
malwares
that
are
basically
hiding
right
there.
They
are
inside
of
your
container
image
and
they
are
hiding
using
either
encryption
or
in
some
cases
they
are
using
techniques
like
tar
files,
seven
layers
of
style
files
right,
so
they
are
hiding
inside
of
your
container
image
and
what
they
are
doing.
B
They
are
waiting
for
you
to
execute
the
container
image
in
order
for
them
to
basically
run
the
malware
and
decompress
themselves
and
start
attacking
your
organization
and
the
technique
for
adding
an
evasive
malware
into
your
container
image
is
something
called
a
supply
chain
attack.
So
what
does
it
mean?
It
means
that
you
as
a
developer,
you
are
building
a
container
image.
You
are
leveraging
open
source
components.
You
are
leveraging
public
container
images
and
inside
of
those
open
source
components,
there
might
be
an
evasive
malware,
meaning
an
attacker
got.
B
What
we
have
seen
in
the
in
the
wild,
we
actually
have
seen
cases,
and
this
is
a
good
example.
You
can
see
a
screenshot
of
it
where
images
that
do
not
have
any
vulnerability.
When
you
scan
them,
you
do
not
see
any
malware.
Basically,
those
images
have
evasive
malware
and,
as
an
example,
we
have
honeypots
in
the
internet
and
in
those
honeypots
we
have
weak
configurations
of
ma
of
kubernetes
and
docker,
and
what
we
have
seen
we've
seen
attackers
trying
to
attack
us
by
running
legitimate
container
images.
B
A
A
A
A
A
C
A
A
A
We
need
something
to
observe
how
the
container
is
behaving,
and
this
is
not
your
ordinary
debugging
troubleshooting
tool
that
you
may
have
used
from
other
contexts,
because
now
it
needs
to
be
able
to
handle
malware
and
malware
may
be
evasive.
Maybe
tricky
amir
gave
a
few
examples
for
how
it
can
be
evasive,
so
we
need
something
that
is
very
robust
and
very
comprehensive,
and
luckily
for
us
we
have
just
the
perfect
technology
to
build
something
like
that
on
which
is
evpf
in
linux.
B
A
Running
in
a
container
in
user
space,
on
top
of
the
container
runtime
on
top
of
the
operating
system,
api
and
so
on,
eventually,
it
will
make
some
call
into
the
kernel.
This
is
how
operating
systems
work
and
we
will
be
down
there
waiting
for
this
to
happen
and
collect
this
information
and
ebpf
is
just
what
gives
us
the
opportunity
to
instrument
these
critical
parts
of
the
kernel
and
extract
the
information.
A
B
A
A
And
this
is
what
I
will
be
using
now
for
the
demo,
so
actually,
let's
switch
gears
into
the
demo
all
right.
So
what
I
want
to
do
for
the
demo
is
work
you
through
a
few
examples
for
patterns
that
we've
seen
in
malware,
and
especially,
I
want
to
show
you
how
it
looks
like
from
the
operating
system
point
of
view
and.
C
B
B
A
A
A
A
And
finally,
the
file
change
mode
at
system
call
which
is
used
by
geomod
binary
in
order
to
make
the
permission,
change
and
I'm
going
to
also
filter
the
output
from
tracy,
because
it's
going
to
be
a
little
bit
noisy
for
only
things
that
originate
from
this
shell.
Here,
because
I
want
it
to
be
more
readable.
A
A
A
C
B
A
A
Because
we
are
writing
to
the
file
actually
here,
so
these
four
events
here
are
what
copies
from
the
socket
into
the
file
you
can
see.
The
arguments
here
is
the
path
to
the
file
and
some
sizes.
I
guess
it's
for
it's
four
events,
because
the
download
is
being
done
in
chunks,
so
we
have
four
chunks
once
wget
is
finished
downloading
the
file
zesh
is
executing
the
change
mode,
executable.
A
A
C
A
A
C
A
A
A
A
A
A
A
B
A
A
A
A
B
A
For
example,
this
one
this
memprot
alert
event
is
not
a
system
called
it's
not
a
function,
it's
not
raw
data.
This
is
tracy
trying
to
tell
you
that
hey
they
make
the
protection
change
to
executable.
This
is
interesting.
You
should
note
and
again
tracy
here
is
telling
you
hey
a
memory,
mapped
region
with
write
and
execute
permissions.
A
B
B
A
A
A
C
A
C
A
B
C
A
A
A
For
example,
amir
mentioned
supply
chain
attacks,
let's
think
of
a
scenario
when
we
need
some
kind
of
utility
and
we
found
the
solution
as
a
perfect
container
on
docker
hub.
Let's
say
that
we
were
looking
for
a
web
service
that
returns
the
current
date
and
time,
for
example.
So
I've
created
something
like
this
here.
A
A
But
what
I
don't
know
is
that
it
actually
has
ran
a
malware
on
my
servers
and
it
has
used
all
of
the
evasion
techniques
that
we've
just
discussed
actually
a
few
others
that
we
didn't
in
order
to
evade
us
or
any
tools
that
we
were
using
and
if
we
were
to
scan
this
image
using
any
static
image
scanning
tool.
So,
let's
use
for.
C
C
A
A
A
A
Your
needs,
we
just
run
it
in
this
sandbox
and
we
collect
the
output
and
we
show
you
this
lovely
dashboard.
That
summarizes
all
of
the
information,
but
the
real
interesting
thing
here
is
not
the
pretty
dashboard.
The
really
interesting
thing
here
is
that
we
have
took
all
of
the
experience
that
our
research
team
has
from
studying
malware
and
we
have
created
a
an
extensive
library
of
signatures
that
we
think
are
interesting,
for
you,
for
example.
Here
is
the
fireless
attack
by
the
way.
A
This
is
this:
the
scan
result
for
the
same
container,
the
rotten
date
container
that
we've
just
seen.
So
we
see
here
the
file
is
attack,
and
this
is
the
line
we
see
here.
The
upx
executing-
and
here
is
the
evidence
we
see
here-
something
else
that
we
didn't
see
before
that
this
id
address
might
be
might
be,
have
a
bad
reputation.
B
A
A
B
Thank
you.
So
to
summarize,
we
are
talking
today
about
a
party
that
we've
identified
in
our
cluster
and
the
fact
that
static
scanning
that
are
looking
for
vulnerabilities,
malware
misconfigurations
cannot
address
the
issues,
because
this
part
is
basically
because
of
an
invasive
malware.
We
also
talked
about
the
fact
that
runtime
security
can
help
you,
because
the
runtime
security
will
detect
and
prevent
malicious
activity,
but
the
reason
need
to
shift
left
the
reason
to
find
the
bad
behavior
the
invasive
malware
before
you
go
production,
and
this
is
what
we
presented
today.