►
From YouTube: Data Science for Infrastructure: Observe, Understand, Automate - Zain Asgar & Natalie Serrino
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Data Science for Infrastructure: Observe, Understand, Automate - Zain Asgar & Natalie Serrino, New Relic
Cloud-native applications today are increasingly complex and therefore increasingly hard to understand. It’s critical to connect decisions around resource allocation and architecture to business metrics such as end-user latency, but very difficult to do in practice. Ultimately, understanding how your systems behave and why is a data analytics problem. Like most data analytics problems, the trick is in collecting and wrangling the right data sources.
In this talk, you will learn how Pixie, an open-source observability platform for Kubernetes, can be used to painlessly turn low-level telemetry data into high-level signals about system health. The talk will also show these high-level signals can be used as input to a variety of use cases, such as detecting SQL injections and K8s deployment autoscaling.
A
Hey
everyone
we're
going
to
talk
about
data
science
for
infrastructure,
so
three
steps
we
think
are
the
three
steps
observe
understand
and
automate
before
we
get
started
just
a
really
quick
overview.
I'm
zane-
and
this
is
natalie-
I'm
a
gm
at
new
relic
and
formerly
the
co-founder
and
ceo
of
pixie
labs.
A
We
got
acquired
by
new
relic
and
both
of
us
are
working
on
the
pixie
project.
Now
and
natalie
is
a
prince
engineer
on
our
team,
so
we're
both
machine
learning
people
by
my
background
and
have
basically
spent
the
last
decade,
building
scalable
data
and
machine
learning
systems,
and
we
really
wanted
to
try
to
understand
how
to
apply
our
our
knowledge
from
that
space
into
observability
and
infrastructure.
A
So
when
we
started
you
know
in
the
space,
we
started
to
think
that
this
is
just
another
data
problem.
It's
easy
for
machines
to
really
you
know
very
quickly,
generate
gigabytes
of
data,
but
it's
really
hard
for
us
to
actually
get
complete
coverage,
especially
in
distributed
environments
where
it's
hard
to
instrument
all
your
applications.
A
A
A
In
reality,
most
of
the
time
is
actually
spent
over
here,
even
though
a
lot
of
coverage
is
given
to
the
next
two
steps.
We'll
talk
about
so
the
next
thing
you
typically
do
after
you
gather
the
data.
Is
you
do
some
kind
of
data
transformation
and
this
actually
gets
the
most
disproportionate
emphasis
a
lot
of
times.
People
talk
about
machine
learning
models
and
all
sorts
of
sophisticated
tools
that
can
build
over
here.
A
You
know
and
something
over
here
could
be
a
simple
statistical
machine
learning
model.
It
could
be
a
regular
expression,
it
could
be
aggregates,
there's
a
whole
gamut
of
things
that
it
could
actually
be,
and
the
goal
over
here
is
to
be
able
to
transform
the
raw
data
into
some
signal
that
can
be
usable
downstream.
An
example
of
this
is,
you
basically
take
let's
say,
cpu
information
for
a
bunch
of
different
replicas
of
your
application
running
and
you
say,
oh,
like
look.
A
So
you
know
it's
pretty
common
to
be
able
to
generate
limits
and
alerts
with
existing
tools,
but
one
of
the
things
that
we've
been
pretty
excited
by
and
we'll
talk
about
over
here
is
the
possibilities
by
using
the
kubernetes
api
to
really
connect
the
first
two
steps
and
then
close
the
loop,
and
by
doing
that,
you
can
ultimately
take
data.
All
the
way
to
the
entire
pipeline
have
kubernetes,
go,
adjust
the
actual
application
parameters
and
then
go
back
into
the
cycle.
Gather
data
transform
it
and
automate.
A
So
there
are
a
lot
of
steps
over
here
and
I
don't
really
want
to
go
into
the
details
of
exactly
what
types
of
metrics
or
what
types
of
information
is
relevant,
but
just
quickly
for
raw
data.
Typically,
people
are
looking
at
logs
metrics
and
requests
for
transformations
you're,
looking
at
aggregates
anomaly,
detection,
regular
expressions
and
on
the
the
signal
based
stuff
you're.
Basically
looking
at
trying
to
update
some
systems
or
be
able
to
do
things
like
allocate
more
resources,
there
are
a
bunch
more
things
you
can
do.
A
A
What's
going
on
and
then
automate
workflows,
so
fixie
was
built
to
solve
these
problems.
We're
gonna
have
three
core
principles
that
you
follow.
The
first
one
is
to
try
to
do
as
much
automated
telemetry
as
possible,
and
we
utilize
this
new
kernel
technology
or
maybe
a
relatively
new
kernel
technology
called
ebpf,
and
there
are
other
talks
that
talk
about
this.
I'm
not
going
to
go
to
details,
but
in
a
nutshell,
it
kind
of
allows
you
to
go
insert
code
dynamically
into
the
kernel
to
be
able
to
observe
applications.
A
So
just
a
quick
overview
of
the
features
or
feature
set
of
the
project.
Yeah,
I
think
I
forgot
to
mention,
but
pixi
is
an
open
source
cnc
of
sandbox
project,
so
you
can
check
it
out,
but
in
terms
of
auto
telemetry
using
ebpf,
we
can
capture
application
network
and
infrastructure
data.
This
includes
things
like
application
profiles
and
full
body
requests,
and
we
have
pretty
low
overhead.
A
So
we
really
think
about
this
as
infrastructure
as
code,
we
can
basically
write
code
to
to
be
able
to
generate
all
the
all
the
metrics
we
need
to
automate,
and
then
this
can
be
accessed
to
the
api.
So,
instead
of
pixie,
everything
is
a
script.
Our
ui
actually
runs
the
script
as
well
and
we
can
easily
integrate
in
the
downstream
tools
through
our
api
and
they're,
pretty
active,
grafana,
plugins
and
plugins
for
other
sas
products
for
pixie.
A
The
second
is:
we
wanted
to
be
able
to
automate
some
more
data
collection
that
we
didn't
think
about
a
priority,
and
the
third
thing
is:
we
really
did
not
want
to
invent
a
new
language,
because
you
know
everyone
starts
off
starting
to
write
a
new
language
and
it
gets
very,
very
complex
for
for
users
to
use.
So
you
can
actually
see
a
little
snippet
of
pixel
code
on
this.
I
guess
on
the
right
side
for
everybody
and
that
particular
code
and
not
to
go
into
the
details,
but
it's
based
on
a
python.
A
It's
based
on
python,
it's
actually
100
compiled
python
and
it's
based
on
this
library
called
pandas,
which
is
for
data
processing
and
it's
very
commonly
used
in
data
science,
and
you
can
basically
see
over
there
we're
saying.
Oh
look
at
all
the
http
events
for
the
last
30
30
seconds
and
then
capture.
What
pod
information
is
there
and
then
you
know
pick
out
some
specific
columns.
A
I
don't
really
want
to
watch
the
details
of
the
language,
but
you
know
it
basically
gives
like
a
pretty
simple
way
to
work
with
data
in
your
system,
so
just
to
recap,
pixel's
an
embedded
dsl.
So
it
is
still
like
a
domain
specific
language.
It
doesn't
work
like
generic
python,
but
it
is
valid
python
and
in
particular
it
is
valid
pandas,
which
is
very
common
in
data
science
and
it's
built
for
data
analysis
and
machine
learning.
A
So
we
have
core
capabilities
like
you
know,
running
tensorflow
models
inside
a
pixel
or
being
able
to
run
aggregates
and
joins
and
stuff
pretty
quickly.
In
particular,
you
know
just
to
go
into
one
more
level
of
detail.
Pixel
is
what
people
call
a
data
flow
language,
so
it
specifies
what
is
a
logical
flow
of
data
and
it's
really
pixi's
job,
to
figure
out
how
to
optimize
and
execute
this
so
that
it's
efficient
in
your
system.
A
A
A
So,
to
recap
you
know:
pixel
provides
an
interface
to
work
with
data.
It
allows
us
to
construct
powerful,
composable
workflows,
and
the
next
step
over
here
is
that
we
have
a
couple
of
demos
that
will
actually
demonstrate
those
capability
that
natalie
will
walk
through.
So
the
first
one
is
doing
slack
alert
on
sql
injection
attacks,
so
it's
like
we'll
not
only
will
talk
to
what
sql
injection
is,
but
if
there's
a
sql
injection
attack
in
your
cluster,
we'll
be
able
to
flag
that
and
generate
a
slack
alert.
A
A
B
Great
thanks,
zane
okay
sounds
like
this
mic
works
so,
as
he
said,
I'm
natalie
I'm
going
to
be
taking
you
through
some
of
these
demos
that
are
kind
of
meant
to
be
proofs
of
concept
to
illustrate
some
of
the
principles
he
was
talking
about.
Before
about
how
we
can
automate
more
of
our
workflows
using
the
raw
telemetry
data
that
exists
in
all
of
our
applications.
B
First,
we
would
need
to
deploy
pixie
to
a
cluster.
This
takes
about
three
minutes,
so
I've
created
a
video
to
kind
of
fast
forward
that
for
you,
but
I
think
what
we're
trying
to
illustrate
here
is.
You
know
how
we're
trying
to
set
a
path
where
it's
much
easier
to
collect
data
with
ebpf.
Instead
of
spending
a
lot
of
time
manually
instrumenting
your
application,
you
can
just
run
something
like
pxdeploy
or
other
ebpf-based
tools
and
automatically
start
collecting
this
raw
data
from
your
cluster.
B
B
B
B
B
Now,
I'm
actually
going
to
use
pixi
in
order
to
you
know
see
that
this
query
has
been
executed,
and
this
is
a
very
simple
view
in
pixi,
but
it
basically
collects
all
the
mysql
data
on
your
cluster.
So
I
guess
I
should
say
that
we've
deployed
this
dvwa
application
to
our
kubernetes
cluster.
So
that's
how
pixi's
collecting
this
data.
B
B
B
B
This
would
be
really
bad,
so
I'm
going
to
talk
about
sql
injection
detection,
which
we
think
is
really
important,
because
no
application
is
perfect
and
free
from
vulnerabilities,
but
it's
worth
noting
that
prevention
is
really
really
important
when
it
comes
to
sql
injections.
So
detection
is
only
part
of
the
story.
We
really
want
to
be
as
application
developers
thinking
about
security
and
doing
prevention
as
a
first
step,
but
let's
say
that
we
have
an
application
out
in
the
wild
and
we
want
to
make
sure
it's
not
running
any
sql
injection
attacks.
B
There's
two
main
ways
that
we
can
do
this
once
we
have
a
list
of
all
the
sql
statements
that
the
code
is
running.
The
first
is
a
rules-based
approach.
You
might
say,
I'm
going
to
parse
all
my
sql
queries
and
make
sure
that
it's
not
actually
using
any
syntax
that
I
have
deemed
prohibited,
such
as
unions,.
B
B
Another
way
you
can
do
it
is
using
machine
learning.
You
can
train
your
model
on
real
world
examples
of
things
that
are
legitimate
queries
and
things
that
are
sql
injection
attacks
that
could
theoretically
learn
that
using
a
union
in
this
scenario
is
okay,
but
using
it
in
a
different
scenario,
is
not
okay.
B
A
major
complication
with
this
is
where
to
get
the
data
set.
You
have
to
have
a
list
of
data
that
you
know
has
known
sql
injection
attacks
that
reflect
your
application,
so
this
can
actually
be
a
major
blocker
to
doing
something
like
the
machine,
learning
based
approach
and
so
for
this
demo,
I'm
actually
just
going
to
be
doing
a
rules
based
approach
with
regex's,
but
as
the
instead
pixel
supports
running
tensorflow
models.
B
So
if
you
were
to
have
the
data
set
available
to
train
a
model
on
detecting
sql
injection
attacks
using
machine
learning,
you
could
easily
you
know
plug
that
in
instead
of
the
regex
based
approach
just
want
to
call
out,
in
addition
to
dvwa
a
tool
that
is
helping.
This
demo
is
sql
map.
It's
a
really
cool
command
line
utility
that
basically
tries
to
attack
your
database
and
figure
out.
Are
there
vulnerabilities
in
it
that
exist,
and
so
we're
going
to
be
using
that
today.
B
First
off
we
want
to
have
the
script
that
actually
detects
the
sql
injection
attack.
So
what
this
script
is
doing
right
here
is
it's
actually
just
showing
you
hey?
What
are
the
sql
queries
that
exist
in
my
cluster,
but
we
actually
want
one
that
runs
pattern
matching
to
detect
what
one
could
be
a
sql
query
or
a
sql
injection.
B
B
B
B
B
B
So
basically,
this
script
has
successfully
given
us
a
list
of
these
injections.
Now
some
of
these
might
be
legitimate
queries,
but
you
know
we
just
want
to
flag
things
that
are
breaking
the
rules
that
we've
said
aren't
allowed.
So
what
we
can
see
right
here
is
that
someone
wrote
a
very
someone.
Aka
sql
map
wrote
a
very
suspicious
looking
query
right
here.
B
B
And
if
the
demo
gods
are
with
me,
we
should
see.
Oh,
it's
detected
a
possible
sql
injection.
So
this
thing
has
told
me
that
the
you
know
a
symbol
or
I
guess
it
just
always
runs
this
one.
First,
unmatched
single
quote:
it's
running
this
one
a
few
times,
and
so
what
we
can
see
now
is
that
I
have
a
full
data
pipeline
that
is
looking
at
my
sql
data
and
pinging
me
on
slack
whenever
it
detects
a
possible
injection.
B
So,
to
recap:
we've
basically
applied
the
same
thing
that
zane
has
talked
about,
which
is
we
use
pixi
to
gather
raw
data
and
transform
it
into
a
signal
by
collecting
these
raw
sql
events
and
using
a
script
to
try
to
use
regexes
to
establish
which
ones
might
be
sql
injections
and
then
we're
doing
something.
Based
on
the
signal
which
is
pinging
slack,
because
we
see
that
there's
been
a
problem
all
right.
Let's
move
into
the
next
demo
for
a
completely
different
use
case,
which
is
basically
workload
scaling.
B
B
B
B
B
Maybe
I
want
to
look
at
the
latency
of
my
downstream
dependencies.
Maybe
I
want
to
look
at
my
number
of
outbound
connections.
Maybe
I
want
to
look
at
application
specific
metrics
like
how
long
this
particular
function
is
taking
there's
so
many
things
that
we
could
use
in
order
to
drive
the
auto
scaling
of
the
deployments.
B
Also,
all
of
these
slides
are
are
par,
are
uploaded
to
the
session.
So
if
you
want
to
look
at
them
for
reference
later,
they're
available,
kubernetes
is
amazing.
It
has
such
an
api
driven
approach
and
it
makes
auto
scaling
based
on
these
complex
factors
possible
right
off
the
bat
it
supports
both
horizontal
and
vertical,
auto
scaling.
Horizontal
basically
increases
the
number
of
pods
in
a
deployment
vertical
increases.
B
Something
like
cpu
or
memory
applies
to
everyone,
but
you
may
have
something
very
specific
for
your
application
that
you
need
to
scale
on
the
custom.
Metrics
api
is
really
useful
because
you
can
hook
in
your
own
metric
into
the
auto
scaler
to
add
more
pods
or
add
more
resources
to
a
pod,
we're
going
to
be
using
a
very
sophisticated
demo,
app.
B
B
That
was
meant
to
be
a
joke
people
anyway.
You
know
the
kubernetes
sig.
Did
this
amazing
basically
like
demo
app
of
how
to
make
a
custom
kubernetes
metrics
api
server?
So
we're
really
appreciative
of
the
work
they
did
in
putting
that
together,
because
we
based
this
example
off
that
api
server?
If
you
want
to
build
your
own,
you
should
totally
use
this
project
as
a
starting
point.
B
B
B
B
The
metric
provider
that
we
have
is
pretty
simple.
It
basically
takes
this
pixel
script,
which
computes
the
average
number
of
http
requests
by
pod
over
the
time
window
and
it
fulfills
these
endpoints.
These
are
defined
on
the
custom
metrics
api
get
metric
by
name
and
get
metric
by
selector
and
list
all
metrics.
So
you
can
see
that
in
just
about
200
lines
of
code,
I've
easily
created
a
metric
server
that
will
auto
scale
my
request
based
on
or
auto
scale
my
deployment
based
on
the
number
of
requests.
B
So,
to
summarize,
what
we
just
saw,
we
collected
the
raw
http
requests
in
pxe,
although
you
could
do
this
with
anything
that
taps
into
your
http
requests.
We've
calculated
the
request
per
second
by
pod
and
we've
plugged
that
into
the
kubernetes
api
to
auto
scale
or
deployment,
I'm
passing
it
back
off
to
zane.
Thank
you.
A
Awesome
well
thanks
for
those
demos
natalie,
so
in
these
demos,
natalie
basically
showed
how
you
can
take
some
simple
data
run.
Some
simple
data
workflows
on
pixi,
in
particular
for
the
sql
injection
there's
a
blog
post
on
pixi's,
open
source
blog.
That
goes
into
a
lot
more
detail
if
you're
interested
on
it
and
then
we're
going
to
be
posting
a
blog
on
the
auto
scaling,
auto
scaling
project
next
week,
which
will
have
a
lot
more
of
the
technical
details
about
how
this
works
and
code
that
you
can
look
at.
Oh.