►
From YouTube: Lightning Talks: Detecting Data Exfiltration on the Edge with Pixie - Zain Asgar, New Relic
Description
Lightning Talks: Detecting Data Exfiltration on the Edge with Pixie - Zain Asgar, New Relic
Detecting data exfiltration in your Kubernetes cluster is important but hard. Capturing the right data, especially encrypted data, in order to perform the analysis can be a hassle. Additionally, it can be a non-starter to export sensitive requests outside of the cluster to perform this analysis. In this lightning talk, you’ll learn how Pixie (an open source, CNCF sandbox project), can be applied to attack this problem. Pixie’s auto-telemetry, in-cluster edge compute, and scriptability make it a powerful tool for anyone looking to identify data exfiltration attacks in their cluster. We’ll show a demo which will also be open source for attendees to reference later.
A
A
B
Hi
everyone
today
I'll
talk
about
data
exfiltration
on
the
edge
with
pixi.
So
a
little
bit
about
me,
I'm
zane
and
the
general
manager
and
vp
at
new
relic
working
on
the
pixi
and
open
ecosystem
teams.
I
was
originally
the
co-founder
and
ceo
of
pixie
labs,
which
was
acquired
by
new
relic
and
I'm
also
an
adjunct,
professor
of
computer
science
at
stanford.
B
B
Our
goal
is
to
demonstrate
some
ideas
and
start
some
discussions
not
to
try
to
push
this
towards
a
production
use
case.
So
what
is
the
data
exfiltration
risk?
So
you
know
we
think
that
data
exfiltration
is
a
huge
risk,
and
that
means
that
you're
leaking
in
information
outside
of
your
cluster.
So,
for
example,
you
might
be
sending
credit
cards,
social
security,
numbers,
phone
numbers
and
other
other
identifiable
information
outside
of
your
kubernetes
cluster,
or
even
within
your
services,
where
it
should
not
be
happening.
B
And
ultimately
you
know
this
can
come
back
and
and
cost.
You
know
in
forms
of
probably
frozen
money
because
of
all
the
data
loss
and
potential
customer
trust
loss
that
you'll
have
so.
The
question
right
here
is:
wouldn't
it
be
great
if
sensitive
data
leaving
your
cluster
could
be
found
in
a
in
a
transparent
way?
And
you
know
we
say
that
it
all
starts
with
observability,
mostly
because
pixi
is
originally
a
performance,
observability
tool
and
we're
trying
to
extend
it
to
do
some
more
use
cases.
So
what
is
fixie?
B
Probably
not
a
lot
of
folks
in
this
community
are
familiar
with
that,
but
we
started
out
with
our
goal
of
performance
debugging
without
manual
instrumentation,
so
we
do
all
the
basic
stuff,
like
cpu
memory
network,
you
know,
grab
all
message,
fans
and
latencies
and
you're,
probably
seeing
where
this
is
going
now
and
things
like
performance
profiles,
but
the
three
characteristics
of
pixi,
which
I
think
make
it
very
useful
to
do
things
in
the
security
space.
So
the
first
one
of
them
is
this.
B
This
concept
of
zero
instrumentation
there's
another
session
going
on
next
year
in
ebpf,
but
pixie
basically
leverages
ebpf
to
allow
you
to
monitor
your
applications
without
doing
any
manual
instrumentation,
and
this
level
of
instrumentation
is
pretty
deep
and
not
only
allows
you
to
capture
things
like
here's,
the
http
message
data,
but
it
can
tell
you
you
know
what
commands
are
getting
executed,
what's
actually
contained
in
those
message
bodies,
and
it
can
do
all
of
this
without
having
to
change
your
application.
So
you
know
that
the
observability
always
exists.
B
The
other
characteristics
we
think
is
useful
is
that
we
have
a
distributed
architecture,
which
means
that
we
can
take
a
look
at
a
lot
of
data.
Since
you
can
deploy
this
on
every
single
node,
you
don't
have
to
worry
about
having
bottlenecks
for
looking
at.
What's
for
actually
inspecting
the
data,
and
the
third
thing
is
this
concept
around
the
scriptable
interface.
You
can
actually
write
scripts,
which
can
look
for
which
can
look
for
your
data
loss
so
very
quickly.
B
I'll
talk
about
you
know
very
high
level
diagram
of
the
pixy
architecture,
so
at
the
highest
layer
you
know
you
have
our
apis,
uis
and
clis.
There's
a
cloud
system
to
help
orchestrate
all
this,
but
most
of
the
heavy
lifting
happens
down
here
with
the
collector
and
aggregator
and
the
actual
kubernetes
nodes
we
deploy.
This
thing
called
the
pixie
edge
module
on
every
single
node.
There's
a
data
collector
based
on
ebpf
that
collects
information
across
all
your
pods
running
on
kubernetes,
and
then
we
store
this
information.
B
You
know
in
like
a
ring
buffer,
so
we
can
query
it
later
and
run
analysis
on
it
and
everything
inside
of
pixie
descriptable,
with
a
language
designed
to
do
data
analysis
and
machine
learning.
So
it's
basically
valid
python
valid
pandas
and
you
can
essentially
operate
on
data
frames.
I'm
not
going
to
go
into
too
much
detail
about
this.
There's
more
information
available
available
on
our
website
or
github
repo.
B
B
So
here's
the
demo
scenario
and
we're
almost
obviously
none
of
the
talk.
So
we
have
a
kubernetes
cluster
running
a
legitimate
pod.
That's
making
ssl
requests
to
the
stripe
api.
Then
we
have
two
malicious
fonts
one
of
those
making
http
requests
to
the
post
test,
server
and
another
malicious
bot.
That's
making
https
requests
and
we're
gonna
basically
try
to
see
if
you
can
find
that
in
our
kubernetes
cluster.
B
If
you
go
down
here,
we
have
a
list
of
our
our
namespaces
and
I'm
running
this
data
exfiltration
demo
over
here.
You
can
see
that
there's
some
legitimate
stripe
egress
going
to
some
ip
and
then
there
is
some
malicious
egress
pods
talking
to
some
other
ip.
But
this
isn't
entirely
that
useful.
It's
just
telling
us
there's
some
communication
happening.
B
I
think
I
mentioned
earlier
that
pixie
works
on
its
concept
of
scripts,
so
we
have
a
script
over
here
called
egress,
so
I
can
look
for
pii
egress.
This
is
a
beta
script.
So
if
I
run
the
script,
we're
seeing
a
ton
of
http
traffic
going
to
various
ips,
we
can
turn
on
some
better
dns
resolution
because
pixi
traces,
dns
traffic,
and
with
that
we
should
be
able
to
see
that
there's
some
http
traffic
flowing
to
stripe
and
then
this
yellow
bar
means
there's
mixed
traffic
between
http
and
https
flowing
to
ptsv2.
B
If
I
go
over
here,
we
should
actually
be
able
to
see
an
example
request.
So
an
example
request:
that's
leaking
the
name,
credit
card
information
and
phone
number
if
everyone's
freaking
out
with
the
data
that
we're
looking
at,
we
do
have
capabilities
to
obfuscate
that
so
you
can't
see
it
in
the
ui,
but
essentially
we
found
the
the
request
and
we
can
we
can
trace
it,
regardless
of,
like
I
said,
oop
sorry,
I
meant
to
go
down
here.
B
B
One
of
the
things
I
want
to
say
is
with
ebpf.
This
is
really
only
the
beginning.
There's
lots
of
other
things
you
can
do
and
other
capabilities
built
in
so
one
of
them.
You
know
things
like
monitoring
file,
accesses
system
calls
process,
execution,
look
at
what
files
are
being
accessed
and
which
information
is
leaking
and
all
of
that
can
be
built
in
at
the
bpf
layer,
and
with
that,
that's
all
I
all
I
have
and
please
check
out
our
website
and
github
for
more
information.
B
B
Yeah,
so
that
really
depends
on
what
level
of
things
you're
scanning
the
performance
overhead
of
actually
collecting
the
data.
Our
target
and
pixie
is
under
five
percent
since
we're
mostly
capturing
the
stuff
at
the
kernel
level,
it's
very
efficient
but
yeah
our
target's
under
five
percent.
We
usually
try
to
keep
it
under
like
two
to
three.
C
B
Two
things
one
is:
you
can
host
everything
on-prem,
so
everything
is
available,
both
the
cloud
slide
and
the
stuff
that's
deployed.
Jumping
back
to
this.
We
actually
don't
send
any
data
to
the
cloud,
even
when
you're
running
scripts,
all
the
data
is
being
run
within
your
cluster
anyways
and
if
you
do
view
stuff
in
the
ui,
it's
end
to
end
encrypted.
So
the
cloud
can't
actually
see
the
data.