►
From YouTube: The Art of Hiding Yourself - Lorenzo Fontana, Sysdig
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
The Art of Hiding Yourself - Lorenzo Fontana, Sysdig
Kubernetes security is an ongoing effort today. In this talk we look at how a hacker would attempt to remain anonymous while compromising a Kubernetes cluster. Seconds after a node or a cluster are compromised, the bad actors start to take measures to make sure their hard work can profit for a while. What do they do? They start hiding their traces. Depending on the attack vector, they will need to hide their traces at multiple levels. They will begin by asking themselves some questions: - Are there are audit log mechanisms? - Kubernetes audit log is enabled? Can I tamper it? - There is deep packet inspection? Can I tamper it? - How to hide processes, containers, tasks to the owners? - There’s any non-conventional place where I can put files? - What about hiding my files in the kubernetes etcd? - How can I hide the network connections I make? In this talk we are going to discuss the broader picture of how the second part of an attack is handled by a bad actor.
A
Hello,
everyone-
this
is
the
art
of
ident
yourself
and
I'm
lorenzo
fontana.
I
work
as
an
open
source
software
engineer
at
cystic,
where
I
am
mainly
focused
on
the
clcf
project
alco,
which
is
the
project
for
container
runtime
security.
I'm
the
author
of
linux
observability,
with
bpf
with
david
and
today
I
want
to
you,
know,
put
the
attention
on
something
that
we
always
do
that
we
never
think
about
which
is
closing
the
doors
right.
A
We
continue
closing
doors,
putting
firewalls
doing
all
the
prevention,
all
the
putting
our
system
in
a
state
where
we
feel
like
it's
secure,
because
security
is
a
feeling
right.
You
always
put
in
place
mechanisms
put
in
place.
You
know
redulify
system,
rootless
containers,
everything
your
power
and
that's
very
good
to
do.
A
They
are
there,
they
might
do
a
lot
of
mess
or
whatever
they
might
spend
their
time.
You
know
getting
all
your
stuff
and
doing
a
big
mess,
but
what
if
they
don't?
It
gets
worse
right
because
let's
say
that
they
just
that
they
are
just
in
and
they
don't
do
anything
they
just
you
know,
sit
in
a
car
and
wait
for
you
to
go
to
sleep.
You
know
it
sounds
scary,
but
that's
what
people
usually
want
to
do
with
your
clusters?
A
A
I
want
to
be
in
a
position
where
I
have
a
cluster
that
I
already
compromised.
I
have
let's
say
a
zero
day
kubernetes
I
am
in
the
cluster
and
my
duty
is
to
minimize
the
risk.
I
don't
want
to
be
discovered
while
maximize
the
profit
of
you
know
getting
access
to
other
machines
in
the
cluster,
and
I
don't
want
to
specifically
be
discovered
because
getting
access
to
the
machine
costed
me
time
and
time
is
money
for
the
hackers
to
do
that.
I
have
to
keep
in
mind
always
two
specific
things:
environmental
awareness
and
persistency.
A
On
linux,
environment
awareness
means
being
aware
of
the
of
three
fundamental
things
processes
now
working
as
storage.
You
will
want
to
create
a
process.
No
one
is
should
be
able
to
see
it.
You
want
to
add
your
network
activity,
because
no
one
should
be
able
that,
while
you're
crypting,
bitcoins
or
any
coin,
you
send
out
the
tokens,
for
example,
and
you
want
to
hide
your
files
so
that
no
one
can
see
them
and
specifically,
you
might
want
to
write
the
files.
You
are
exfiltrating
for
the
machine
if
you
are
expecting
data
from
the
machine.
A
There
are
some
strong
assumptions
in
in
this
talk,
which
are
basically
that
I'm
not
here
to
show
you
how
to
break
into
cluster.
How
to
you
know,
have
a
root
kit
or
a
zero
day.
I'm
just
here
trying
to
create
awareness
on
the
fact
that
once
you're
in
once,
you
have
let's
say
access
to
the
master,
to
hcd
to
everything
you
don't
want
to
be
called.
You
want
to
be
there
as
long
as
possible.
The
company
will
last
10
years.
A
You
want
to
be
there
in
the
machine
10
years
without
them,
even
noticing
to
understand
how
to
hide
yourself.
You
first
have
to
understand
how
administrators
of
the
company
that
you
arriving
in
to
look
at
the
processes,
one
very
common
way
of
looking
at
the
process.
Just
qctl
exec
in
a
container
or
a
cessation
to
a
machine
and
just
do
a
ps
ps,
is
a
very
common
command
that
everyone
uses
and
in
my
demo
later
and
in
all
the
examples,
my
malicious
code
will
be
then
just
a
ping
right.
A
A
I
will
be
using
a
technique
which
is
exploiting
the
fact
that
processes
are
read
from
the
profile
system
usually
commands
like
ps,
stop
lsof
all
the
com,
all
the
standard,
unix
common,
don't
access
the
profile
system
directly
with
a
read
cisco:
they
access
the
profile
system
using
the
read
dear
function
from
gdpc
that
then
uses
the
ciscos
underneath
that
is
also
a
dynamic
library
which
is
subject
to
the
rules
of
the
dynamic,
the
global,
dynamic
linker
using
the
global,
dynamic
clinker.
I
can
replace
the
function
that
ps
top
lsof
use
and
use
my
malicious
function.
A
A
Once
you
have
the
slides
you'll
be
able
to
see
the
link,
the
tool
that
I'm
using
for
writing
processes
is
called
lead
process
either,
which
is
from
an
ex
colleague
called
jaluka
borello
and
lip
process.
I
believe
process
either
has
a
very
good
track
of
being
used
by
hackers
to
hide
cryptominers
in
this
machine.
Just
look
for
lead
process
either.
A
Crypto
miners
on
your
favorite
search
engine
and
you'll
have
a
glance
of
that
id
network
activity.
As
I
said,
since
you
just
use
the
standard,
libraries
again
is
basically
the
same.
Of
course
there
are
most
of
more
advanced
techniques,
techniques
for
riding
processes
and
network
activity
like
I
am
not
talking
about.
You
know
loading
the
kernel
module
to
write
the
processes
from
the
profile
system
or
loading.
You
know
a
program
to
wide
connections
directly.
A
I
am
using
a
more
you
know,
lightweight
approach,
but
I'm
here
to
show
a
concept
to
show
the
concept
that,
even
with
this
lightweight
approach,
the
actual
end
operator
that
manages
the
cluster.
If
they
don't
have
anything
special
in
place,
if
they
just
you,
know,
go
and
kill
ctl,
exec
or
ssh
in
the
machine
and
look
for
the
processes,
they
will
have
no
clue
of
this.
A
A
You
can
add
files
in
network
interfaces
like
let's
say
I
create
a
name,
and
you
know
create
many
network
interface
with
different
names
and
then
just
add
them
using
the
technique
that
I
said
before
or
write
stuff
in
inside
other
binaries.
I'd
stop
in
five
systems
that
are
not
mounted
just
create
a
partition,
and
I
ties
I
didn't
want
this
talk
to
become
three
hours,
so
I
had
to
choose
environment
awareness
on
kubernetes
is
even
more
important
than
on
linux,
because
you
have
many
many
components.
A
A
A
A
I
will
use
a
trick
of
deleting
the
pod
entry
from
met
cd
to
id
the
entry
from
qctl
get
box,
because,
basically,
if
I
hide
my
process
and
you
enter
the
machine
and
you
can
see
the
the
process
like
the
ping
process
and
then
you
qctl
getpods,
you
know
see
my
my
container.
I'm
done.
I
can
start
cryptomining
or
whatever.
A
Before
we
go
forward
and
see
the
example
that
I
prepared,
I
want
to
just
do
a
recap
of
what
we
will
be
doing.
We
will
be
creating
about
the
netsi
cooperatives
manifest.
We
will
be
in
a
master
in
a
master
where
we
have
access
to
it,
see
kubernetes
manifest
demand.
The
master
will
also
have
the
folder
at
c,
see
lds,
so
preload
the
file
lc,
lds
or
preload,
and
we
are
able
to
connect
with
cd
for
deleting
pods.
A
Of
course,
all
those
reactions
that
we
do
can
be
seen
from
a
kernel
perspective,
so
I
will
also
be
showing
how
to
use
falco
to
see
those
activities
right
after
I
do
the
I
do
the
the
hiding.
So
I
will
be
just
you
know,
hide
myself,
and
then
I
will
see
how
falco
complains
about
myself,
hiding
in
that
moment
demo
time.
A
Our
demo
is
starting
from
the
evo
pod.
The
evil
pod
is
my
id
inverter.
My
attack
vectors
to
stay
hidden
in
this
evil
pod.
The
first
thing
that
I
have
is
an
init
container,
which
has
access
to
the
os
network
access
to
the
os
process.
Namespace
this
spot
contains
a
base64
encoded
version
and
zipped
version
of
the
library
lib
process
either
modified.
I
the
specific
process,
which
is
called
ping.
A
This
is
basically
the
code
that
library
contains.
I
will
be
linking
it
in
the
slides,
the
library
just.
I
just
changed
the
process
to
filter
with
ping
so
that
when
the
library
goes
and
lists
all
the
processes
for
the
process
comm
and
or
for
top,
it
will
not
go
directly
to
showing
the
process.
If
the
process
is
named
ping,
how
the
library
works,
it
basically
just
loads
itself
before
gbc
and
wraps
the
read
their
calling
here.
A
I
have
the
library
I
uncompressed
the
library
in
user
local
lib
process
cider
so
and
in
particular
I
do
it
in
a
way
that
I
copied
in
the
host
using
the
proc
by
system
every
process
as
a
process
id
under
proc.
Every
process
id
under
proc
has
a
folder
called
root.
That
folder
contains
the
full
file
system
of
the
the
process,
the
full
snapshot
of
the
default
system
of
the
process,
the
view,
so
I
can
just
go
and
load
these.
A
This
is
one
of
the
three
parts,
so
this
part
does
three
things:
first,
it
loads
leap
process
either
in
the
host,
so
that
when
this
is
loaded,
you
can
see
anymore
my
evil
action,
which
is
just
appearing
to
the
club
for
dns.
As
I
said,
the
second
thing
it
does
after
that
is
that
it
register
itself
and
continuously
removes
the
pod
itself
from
hcd
by
removing
the
key
from
lcd
direct,
which
is
registry
power,
defaultival,
podca
control
plane
every
five
seconds,
and
this
is
done
by
connected
to
xcd
using
the
same
technique.
A
I
decided
to
create
everything
under
manifest,
instead
of
using
instead
of
putting
my
pod
in
there
with
the
cube
api
server,
because
in
this
way
I
don't
have
to
deal
with
the
hiding
of
my
deployment
or
a
demon
set.
I
just
add
the
pod
itself
and
I
also
don't
have
to
deal
with
the
fact
that
the
request
of
inserting
the
pod
goes
through
the
api
server
which
might
log
stuff
in
the
when
it
is
audit,
log
or
other
places.
A
A
A
A
A
So
at
this
point
I
would
an
attacker
be
able
to
to
remove
these
the
first
thing
that
they
might
notice
if
they
have
some
kind
of
software
that
you
know.
Let
them
know
that
someone's
writing
under
a
dso
preload
is
that
they
might
want
to
remove
the
preload.
So
at
this
point
they
are
able
to
see
ping
again,
but
unfortunately
they
don't
know
what
is
doing
this
yet
because
I
left
chris
ctl
behind
it
could
have
masked
myself
from
there.
A
A
This
is
the
falco
sidekiq
ui
falco
sidekick
is
a
tool
that
aggregates
all
the
event
from
falco
and
shows
them
in
different
outputs,
in
this
case
it's
using
the
web
ui
so
that
we
can
see
it
through
the
browser
and
I
filtered
the
errors,
because
this
is
a
actually
a
critical
log,
an
alert
and
this
showing
that
someone
opened
up
for
writing
under
hcdc
folder,
and
this
is
just
a
cool
with
the
evil
body
ammo.
This
looks
suspicious,
so
this
was
actually
cool.
A
A
You
can
see
things
from
different
angles.
The
way
the
explorer
worked
was
to
hide
myself
from
ps
that
uses
gdbc
to
do
cs
calls.
If
I
do
the
cisco
directly,
let's
say
that
if
ps
did
the
cisco
directly,
I
wouldn't
be
able
to
hide
myself
from
ps
unless
I
actually
act
the
kernel
and
change
the
code
in
the
kernel
for
the
psc
score.
A
So
since
falco
looks
at
the
ciscos
directly
directly,
falco
was
able
to
see
it.
On
the
other
hand,
falco
was
also
able
to
see
the
container
falco
was
connected
to
the
kubernetes
api
server
and
also
to
the
audit
log.
So
in
general,
more
data
sources.
You
have
different
things
more.
You
are
able
to
see
if
something
happens.
A
Of
course,
this
doesn't
limit
the
ability
of
the
eventual
hacker
who
want
to
try
to
hide
to
find
the
other
creative
ways
of
hiding
themselves,
but
it
should
give
you
an
idea
of
the
fact
that
they
might
or
not
might
be
able
to
buy
themselves
depending
on
what
you
put
in
place
right.
So
if
you
just
leave
the
door
open,
they
will
be
there.
You
won't
see
them
if
you
close
the
door,
it's
more
difficult
for
them
to
be
there
and
if
they
get
there,
you
don't
see
them.
A
If
you
close
the
door
and
you
also
install
cameras-
maybe
you
see
you
see
them,
but
what,
if
they
have
an
invisibility,
you
know
shield
or
something
you
won't
be
able
to
see
them.
So
you
need
like
a
detector
from
his
invisibility
shields.
So
it's
just
you
know
a
game
of
continuously
catching
up
and
that's
what
we
do.
A
That's
it.
This
was
my
demo.
I
hope
that
I
passed
a
message.
I
hope
that
these
can
help
you
in
you
know,
thinking
about
your
security
and
also
in
thinking
that
in
general,
there's
no
such
thing
as
good
security.
You
just
have
to
play
the
game,
and
thanks
a
lot
for
listening
to
my
telecom,
to
see
you
around.