►
From YouTube: System state clustering using eBPF data
Description
Kubernetes Community Days Bengaluru'21
The field of system observability has been greatly enhanced by the application of eBPF. eBPF generates data at critical points in the execution of a system and that data is used for observation via software like Sysdig and Cilium. I propose to utilize the data generated for system state clustering. This is an application of machine learning to the above data to understand if the system is behaving properly or not. The amalgamation of machine learning and system data generation in real-time would open the doors to a plethora of applications like system state prediction, preventive replacement of system components aided by ML. This talk will take the attendees through an idea of how this could be done.
Slides: https://drive.google.com/file/d/1NsSUZGAtRnllU7RoAzLjh6_TOcmiFr50/view?usp=sharing
A
Good
afternoon
welcome
to
the
presentation
on
system
state
clustering
using
ebpf
data.
My
name
is
sujit
samuel.
I
am
a
principal
software
engineer
at
ericsson
in
the
global,
a
accelerator
unit.
One
of
the
responsibilities
that
I
handle
is
the
system
state
observability.
We
monitor
the
behavior
of
the
systems
to
ensure
consistency
and
security.
A
A
A
A
So
you
can
see,
ebpf
can
attach
itself
to
some
execution
points
in
the
kernel
which
are
called
whenever,
let
us
say,
on
the
left
hand,
side,
we
see
a
storage,
driven
ebp.
Of
course,
any
process
that
reads
or
writes
calls
the
ebpf
program,
just
by
the
virtue
of
the
ebp
program
being
attached
to
a
storage
syscall
on
the
right
hand,
side
we
can
see
the
same
thing
but
on
a
network
event,
so
a
traffic
comes
in
or
a
request
comes
in.
A
A
A
A
You
can
attach
this
particular
uv
program
on
the
function
axis
exactly,
and
this
will
tell
you
how
many
times
that
particular
function
has
been
invoked
by
the
virtue
of
this
particular
epp
of
program,
which
executed
in
the
kernel
space
passing
the
value
into
a
map
that
is
being
read
in
the
user
space.
So
here
you
see
in
the
section
in
the
actual
program,
section
actually
looks
up
the
map
and
sees
whether
the
element
already
exists.
If
it
exists,
then
it
increments.
The
value
if
it
doesn't
exist,
then
it'll
create
the
value
and
update
it
right.
A
This
basically
lands
upon
the
premise
that
things
which
are
similar
tend
to
be
closer
to
each
other
than
dissimilar
things
on
the
right
hand,
side
itself,
you
can
see
that
this
would
be
a
graph
right.
We
see
that
there
are
some
black
points.
There
are
some
red
points
and
there
are
some
green
points
and
there
is
a
very
fine.
A
There
is
a
very
well
defined
line
or
a
or
a
demarcation
between
each
of
the
three
different
points.
Now
this
has
been
achieved
by
an
algorithm
called
k,
means
right,
k
means-
and
if
you
say
three
clusters
right
so
k
means
is
a
machine
learning
algorithm
which
says
that
okay,
I
will
help
you
cluster
your
data
points
into
whatever
clusters
that
you
decide
and
based
on
that
this
k-means
will
actually
run
some
sort
of
algorithm
behind
and
say
that.
Okay,
this
cluster
belongs
to
this.
A
This
point
belongs
to
this
cluster,
and
this
cluster
has
the
centroid
of
the
cluster
as
this
and
each
cluster
will
have
a
centroid,
and
the
distance
from
the
centroid
will
actually
be
determining
the
class
of
that
particular
point
right.
So
clustering
is
essentially
a
machine
learning.
Algorithm
that
helps
us
to
group.
A
So
here
just
to
cover
a
clustering
example:
we
we
know
about
the
dataset,
the
iris
dataset.
The
iris
data
is
a
very
famous
dataset
in
machine
learning
and
almost
all
the
people
who
start
machine
learning
actually
definitely
encounters
data
in
their
in
their
in
their
starting
time.
Right
when
we
we
use
a
data
set
to
do
a
lot
of
machine
learning
trials
ourselves.
A
So
what
is
dataset
contains
is
a
list
of
a
number
of
records
with
different
details
about
a
flower,
the
iris
flower
right
there
there
are,
the
details
are
given
like
petal
width,
sepal
length,
petal
length.
All
these
details
are
given
and
there
are
a
lot
of
records
which
actually
need
to
be
segregated
and
clustered
together
to
understand
the
closeness
of
the
data
points
right
now.
If
you
see
this
graph
that
has
been
projected,
we
clearly
see
that
there
are.
There
are
three
different
data
points
right,
so
there
is
a
green
one.
A
There
is
a
yellow
one
and
there
is
a
kind
of
a
violet
one
that
towards
the
end
towards
this
towards
the
inside
right.
So
this
is
actually
a
very
good
example
of
a
machine
learning
algorithm
called
clustering
running
on
that
particular
data
set
and
helping
us
to
identify
these
three
different
clusters
that
you
can
see
right.
I
picked
up
this
example
from
psychic
learn
a
really
great
clustering,
algorithm
called
k-means.
A
Now,
how
do
these,
how
these
two
terminologies,
that
we
discussed
these
technologies,
that
we
discussed
right,
evp
and
clustering
they
can
come
together
right
ebpf
can
help
us
to
generate
data
points
right.
So
what
all
data
points
do
you
want
to
generate?
You
can
generate
data
point
about
access
of
a
particular
file.
You
can
generate
a
data
point
about
network
requests
that
come
in.
You
can
generate
data
points
about
behavior
of
that
particular
system
as
the
cpu
load,
or
is
how
many?
A
A
There
could
be
an
innate
sense
of
algorithmic
understanding,
which
says
that
which
which
the
clustering
can
provide
as
saying
that.
Okay,
this
particular
data
point
that
has
been
generated
might
not
be
a
good
network
access
right.
You
could
actually,
you
could
actually
help
by
doing
some
sort
of
data.
A
Labeling
yeah,
you
know,
create
some
sort
of
test
scenarios
in
which
you
have
negative
classes
and
positive
classes,
and
then
data
labeling
can
actually
aid
this
particular
machine
learning
process
so
not
essentially
clustering,
but
some
sort
of
labeling
that
can
be
used
to
provide
some
sort
of
input
to
supervised
algorithms.
Now
clustering
is
unsuper
algorithm,
but
you
can
use
this
data
that
is
entered
in
ebpf
to
run
some
supervised,
algorithmic,
supervised
machine
learning,
scenarios,
production
scenario
in
production
scenario.
A
You
can
use
the
model
whatever
you
have
generated,
to
actually
predict
in
real
time
whether
data
point
actually
belongs
to
a
right
behavior
of
the
system
or
wrong
beer
of
the
system.
So
our
our
aim
here
is
basically
to
understand
whether
the
system
behavior
right
now
is
actually
correct,
or
it's
not
correct
right
and
the
idea
that
we
are
discussing
here
is
whether
ebpf
can
help
us.
So
clustering
can
help
us
to
understand
or
segregate
the
data.
Whether
ebp
and
ebpf
can
help
us
to
generate
the
data.
A
Whether
the
combination
of
this
can
help
us
understand
whether
the
system
is
behaving
in
a
proper
way.
That's
what
we
are
discussing.
So
I
believe,
if
we
can
generate
the
data,
clustering
can
help
us
cluster
the
data,
so
just
to
elaborate
on
the
scenario
that
we
discussed
in
the
previous
slide
right.
So
there
are,
there
are
some
classes
of
ubf
programs
right
which
are
going
to
detect
network
which
are
going
to
be
attached
to
network
interfaces.
A
So
this
is
an
xdp
program.
There
is
nbp
socket
filter
program,
so
xdp
basically
stands
for
express
data
path,
so
express
data
path,
basically
means
that
this
is
an
interface
that
is
going
to
be
hit
the
first
whenever
a
network
packet
is
received
right.
So
by
that
what
I
mean
is
at
this
point
in
time,
even
a
struct
socket
buffer
has
not
been
located.
The
skb
struct
has
not
been
allocated
and
it
has
not
been
passed
through.
The
network
stack,
so
it
has
first
hit
the
xdp
xdp
buffer
is
allocated.
A
So
what
happens
is
whenever
a
network
packet
is
received,
and
you
have
an
mbp
of
program
attached
to
an
express
data
path
right.
This
particular
program
will
actually
read
that
particular
packet
and
actually
extract
some
details
from
there
right
like
input.
What
is
the
ipad
that
is
coming
from?
What
is
this?
What
are
the
headers?
What
are
the
header
informations?
What
number
of
bytes
whatever
it
is?
You
can
take
any
details
from
it
and
then
you
can
put
into
this
particular
thing
called
a
map
that
we
already
discussed
now.
A
This
map
sits
in
between
of
the
kernel
space
execution,
as
well
as
the
user
space
right.
So
all
the
details
that
you
have
put
into
the
map
are
actually
accessible
from
the
user
space
now
so
once
once
it
once
the
kernel
or
the
uvpf
program
has
actually
put
into
a
map,
you
have
a
user
space
program
that
actually
reads
the
map
and,
let's
say,
pushes
this
data
to
a
to
a
really
fast
cache
store,
like,
let's
see,
really
fast
store
like
radius.
Let's
say
right
what
happens
once
so?
A
This
data
is
continuously
going
to
reduce
right
and
then
your
machine
learning
platform
actually
could
actually
read
this
particular
data
and
run
clustering
on
it
right
run.
Customer
to
understand
whether
this
particular
network
request
that
is
received
is
actually
good.
Or
is
it
a
valid
network
request,
or
is
it
really
because
we
all
have
some
predicted
or
predictable
behaviors
right
any
time
we
anytime
any
behavior
mostly,
is
predictable
right.
A
So
there
are
time
series
algorithms,
which
could
run
on
this
particular
data
to
understand
whether
this
particular
behavior
of
network
or
network
packet
request
is
really
something
that
we
expect.
If
not,
then,
yes,
we
have.
We
have
aided
the
system
observability
by
actually
classifying
a
particular
network
behavior
as
invalid,
and
that
is
possible
because
we
have
used
this
ebpf
and
custom
together.
So
ebpf
can
help
us
to
understand
the
the
the
real
system.
A
A
So,
on
the
left
hand
side
you
can
see
so
there
are
there
are,
there
is
a
receive
queue
and
there
is
a
transmission
queue
and
right,
and
then
there
is
this
bpf
program
that
is
actually
sitting
on
the
driver
on
the
interface
right
and
this
program
can
actually
do
three
things.
You
know
it
can
pass
the
traffic
up.
The
network
stack,
it
can
actually
redirect
it
or
it
can
just
simply
drop.
A
It
drop
basically
means
that
that
packet
will
never
travels
up
the
stack,
never
loads,
your
system
and
if
that's
the
best
place
to
actually
reject
the
incoming
network.
Now
a
lot
of
ddo
distributed
denial
of
service
attacks
actually
can
be
dropped.
If,
if
the
hdp
program
is
really,
you
know
good
enough
right,
so
what
happens?
On
the
left
hand
side,
you
see
an
incoming
request
right.
The
incoming
request
reaches
the
receive
queue
and
at
that
particular
point
of
time
it
is
actually
intercepted
by
the
http
bpf
program.
A
Right
html
actually
intercepts
the
request
and
then
this
particular
bbp
program
actually
can
write
this
data.
Whatever
data
you
want,
along
with
the
time
time
of
the
request
and
everything
into
a
map,
and
then
the
user
space
action
program
actually
reads
the
map
and
sends
the
data
to
clustering
system
to
ascertain
whether
it
belongs
to
a
valid
cluster,
the
valid
cluster
we
saw,
what
is
a
valid
cluster
right?
We
saw
in
the
iris
example
right
there
are
clusters
of.
Let
us
say
there
are
clusters
of
data
points
which
appear
valid.
A
There
are
clusters
of
data
points
which
appear
not
valid,
and
there
are
clusters
of
data
which
can
downrightly
be
rejected.
So
if
we
find
out
that
the
data
that
is
coming,
which
is
being
populated
in
the
map,
actually
belongs
to
a
down
rate,
you
know
rejected
traffic,
then,
yes,
this
methodology
can
actually
help
us
to
reject
the
traffic
right
away
and
help
us.
You
know,
improve
the
performance
system
in
case
of
things
like
attacks
like
ddos
attacks
or
any
other
attacks
that
we
see.
A
A
This
is
one
more
use
case,
which
is
tied
to
the
observability
of
the
system.
Whether
you
want
to
see
whether
the
system
is
behaving
properly
or
not,
basically
means
that
when
you
are
when
your
system
is
executing,
should
a
particular
program
be
triggered
or
should
a
particular
file
be
accessed
or
not
or
whether,
if
it's
getting
accessed,
is
it
access
getting
accessed
correctly
or
not?
So
for
those
kind
of
things
we
can
actually
attach
evp
of
programs
to
events
right
or
events
right.
A
You
can
create
perf
events,
those
particular
events
are
attached
to
kerner
functions,
so
you
have
a
current
function
that
is
triggered
the
kernel
function,
which
is
triggered
actually
has
ebp
of
program
attached
to
it.
Once
the
evp
of
program
gets
triggered
again,
it
populates
the
map.
We
know
the
map
that
map
is
being
accessed
from
user
space
program.
The
user
space
program
actually
takes
the
details
from
the
map
and
sends
it
over
to
the
clustering
algorithm.
A
The
clustering
algorithm
then
determines
whether
that
particular
file
or
the
executable
that
is
accessed,
is
supposed
to
be
accessed
in
that
way
or
not
are
supposed
to
access
that
particular
time
or
not,
and
then
it
says
yeah,
it's
a
valid
access
is
an
online
access.
So
again
this
will.
This
is
one
of
the
use
cases
in
which
you
again
use
machine
learning
to
actually
understand
whether
the
system
that
is
being
observed
has
has
correct,
behavior
or
not
so
we
saw
in
the
previous
slide
the
network
traffic
check
right.
A
So
an
example
in
this
along
the
summer
lines
could
be
system
performance
degradation
check.
So
what
is
the
system
performance
system
performance
is
nothing
but
a,
but
how
you
expect
your
system
to
actually
perform
during
a
during
any
point
of
time
right
when
it
is
getting
regretted.
When
you
have
a
lot
of
read
failures,
let's
say
you're
a
lot
of
read
failures
or
you
have
a
lot
of
packet
drops
or
you
you
have.
A
lot
of
your
lot
of
indicators
are
being
generated
to
say
that
okay,
my
system
is
not
behaving
properly.
A
Those
kind
of
things
are
definitely
can
be
found
out
in
the
kernel
itself,
so
kernel
when
it
is
trying
to
give
out
some
indicators
like
system
problem
is
getting
degraded
at
those
points
of
time.
Ebv
program
can
help
us
to
actually
take
that
data,
move
it
into
a
central
map
and
then
use
the
space
problems
like
to
read
those
maps
and
say:
okay,
yeah
system
performance
might
get
degraded
at
some
point
of
time
in
the
future.
So
that's
a
time
series
analysis
that
we
can
do.
A
You
can
actually
help
have
machine
learning
algorithm,
which
can
actually
read
the
data
of
the
system,
performance
degradation,
data
that
is
being
generated
by
the
kernel
and
actually
help
us
to
predict.
Okay,
you
know
what
I
have
seen
this
trend
earlier,
and
this
basically
means
that
in
another
10
or
15
minutes,
you
are
going
to
have
a
systems
down
notification
coming
up.
So
please
take
care
of
that.
That
again
leads
to
a
use
case,
like
preventive
maintenance
right.
What
is
why
do
you
need
to
maintain?
A
A
It
could
actually
help
us
to
do
much
better
with
the
system
of
visibility,
use
cases
that
we
are
currently
pursuing
and
we
are
actually
currently
working
on
a
lot
of
such
use.
Cases
yeah
right
and
the
implementation
is
right
in
progress,
wherein
some
sort
of
machine
learning
and
ebpf
can
come
together
and
and
whether
it
can
really
help
us
to
predict
a
system
down
much
much
in
advance
as
compared
to
the
existing
methodologies
or
as
computerization
technologies
that
we
have
right
now.
So
with
this
we
come
to
the
end
of
the
presentation.