►
From YouTube: Deep Network Traffic Observability with Pktvisor and Prometheus - Shannon Weyrick, NS1
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Deep Network Traffic Observability with Pktvisor and Prometheus - Shannon Weyrick, NS1
Observability of network traffic can prove very important to the successful operation of modern applications. The ability to divine key information from the flow of network traffic can provide insight useful for operations, debugging, and security. But efficient analysis and collection of deeply inspected, high throughput traffic is hard… especially as the trend towards globally distributed applications continues. How does one organize a fleet of at-scale agents which can analyze network traffic in real time and send the results to a modern observability stack? pktvisor is a free and open source observability agent designed to address these challenges. Developed by NS1 for their global DNS network, it makes use of real time streaming algorithms to efficiently extract counts, top-k heavy hitters, set cardinality, quantiles and other key information from the various networking layers. In this talk we will outline the challenges above and then work through solving them with pktvisor and Prometheus. We will cover installation with containers, configuration, metrics collection to Prometheus via scrape or remote write, and how to query Prometheus to visualize the results. Finally we will look at the future of the project which adds full remote configuration and fleet management.
A
A
A
Both
of
these
projects
are
free
and
open
source,
and
their
goal
is
what
we
call
dynamic
orchestration
of
business
intelligence
at
the
edge.
If
you
haven't
heard
of
this
term
business
intelligence,
it
means
extracting
actionable
insights
from
these
data
streams,
and
the
goal
of
these
projects
is
to
push
that
process
out
to
the
edge.
A
So
deep
network
observability,
what
do
we
mean
by
that?
So
this
is
this
idea
that
we
can
examine
the
traffic
flows
that
are
happening
between
end
users
and
applications
or
among
the
services
that
make
up
your
application
and
we
can
inspect
that
network
traffic,
and
if
we
can
do
that,
we
can
often
gain
insight
into
helping
us
run.
The
applications
helping
us
operate
them.
A
So
a
little
context
from
ns1,
where
these
projects
are
coming
out
of
so
at
ns1
our
goal
is
to
connect
applications
and
audiences.
That
means
application,
traffic,
optimization
and
one
of
the
ways
we
do.
That
is
with
our
managed
dns
network.
So
we
have
this
global
anycast
network
with
many
pops
around
the
world.
That's
delivering
this,
this
dns
traffic,
and
we
need
to
maintain
this
network
right.
We
need
to
tune
it
and
so
much
like
a
cdn.
A
A
We
are
often
the
subject
to
attacks
right,
malicious
traffic,
ddos
attacks
and
that's
actually
one
of
the
ways
that
pac
advisor
was
originally
created
was
to
give
us
visibility
into
this
type
of
traffic
to
help
us
understand
it
to
help
us
protect
against
it,
and
then
this
visibility,
we're
talking
about,
has
sort
of
two
different
ways
that
that
we
need
to
understand.
One
is
this:
at
the
node
very
high
resolution.
A
So
what
are
the
questions
that
we
want
to
ask
of
this
data?
What
are
these
insights?
We
want
to
pull
well.
It
certainly
includes
simple
metrics
that
you're
all
familiar
with
counters
and
rates
of
things.
So,
of
course,
how
many
packets
have
we
seen
on
a
certain
interface
or
what's
the
rate
of
ingress
or
egress,
but
this
is
where
we
want
to
go
deeper.
A
We
want
to
use
streaming
algorithms
to
do
this
deeper
analysis
and
give
us
better
insight
directly
at
the
edge
here
and
so
one
example
is
frequent
items
otherwise
known
as
like
heavy
hitters
top
10
lists
across
different
network
dimensions.
So
that
that'll
be
things
like
what
are
the
top
10
ip
addresses?
What
are
the
top
10
queries?
A
We
care
about
cardinality
so
again
right
there
on
the
edge.
We
want
to
know
things
like
how
many
unique
ip
addresses
have
we
seen
in
a
certain
time
period
right,
how
many
unique
query
names
have
we
seen
in
a
certain
time
period,
quantiles
and
histograms-
also
very
useful,
of
course,
what's
the
p90
or
p99
of
of
a
transaction
timing?
What's
the
histogram
of
different
response,
payload
sizes
happening
right
now
on
the
security
side,
so
amplification
factor
is
useful.
Are
we
seeing
very
small
queries
come
in,
but
very
large
responses
go
out.
A
Are:
are
people
abusing
that
right
now
for
anybody
who's
familiar
with
dns
at
all?
You
might
have
had
this
problem
before,
which
is
I
deleted?
A
dns
record
who's
still
querying
this
record
that
I
thought
I
deleted
right
and
trying
to
track
that
down,
find
the
sources
and
also
when
we
care
about
the
sources
of
things
we
care
about
what
geographic
regions
are
queries
coming
from
and
what
asn
numbers
are
are
querying
our
network,
and
then
you
know
just
to
summarize
the
security
bit
again.
A
Understanding
whether
traffic
is
malicious
or
legitimate
is
something
we
need
to
do,
and
it's
not
always
that
easy
to
figure
out.
Just
because
we
see
a
spike
in
traffic
doesn't
mean
that
it's
malicious
right,
it
could
be
legitimate,
it
could
be
in
our
case
it
could
be
a
recursive
resolver
farm
whose
cash
has
gone
cold
and
they're
just
sending
a
lot
of
queries
into
us
at
the
moment.
A
But
if
we
do
think
it's
malicious,
we
need
to
understand
what
type
of
attack
it
could
be.
Is
it
a
random
label
attack?
Is
it
widely
distributed
and
that
that
widely
distributed
piece
by
the
way
is
where
this
cardinality
can
help
us?
If
we
see
the
number
of
unique
ip
dresses
blow
up
in
a
very
small
period
of
time,
it
could
indicate
a
distributed
attack.
A
A
A
But
one
of
the
goals
of
the
most
recent
rewrite
is
to
have
a
very
plugable
system,
both
for
the
way
we
get
data
into
it
and
the
way
that
we
analyze
the
data
and
so
there's
a
modular
system.
For
doing
this,
we
can
imagine
many
more
types
of
input
sources.
This
is
where
we
might
start
seeing
things
like
s,
flow
and
net
flow
input,
sources,
dns,
tap,
ebpf,
envoy,
tabs
right.
The
goal
is
to
have
this
pac
advisor
agent
sit
close
to
data
sources
and
be
able
to
passively
observe
using
this
this
tap
methodology.
A
A
It's
able
to
to
do
inspection
on
different
types
of
traffic,
different
protocols
and
certainly,
as
we
get
more
input
sources
that
will
expand
as
well,
but
pac
advisor's
view
is
this
data
that's
flowing
along
the
wire
it's
coming
in
in
waves?
It's
it's
going
up
and
down,
and
it's
able
to
access
information,
that's
inside
the
packets
and
that's
certainly
things
like
the
source,
ip
and
the
destination
ip.
A
But
when
we
say
deep
analysis,
that
means
it's
going
in
and
it's
getting
to
that
application
layer,
information
and
it's
pulling
out
things
that
are
specific
to
applications
right.
So
in
this
case
dns-
and
that
means
query
id
query-
name
number
of
records
that
are
that
it's
returning
the
type
of
query
and
so
forth,
and
so
it's
observing
this
in
real
time
as
the
traffic
is
coming
in
and
it's
summarizing
it's
using
these
streaming.
A
A
So
here's
a
view
of
pac
advisor
and
it's
it's
policy
system
and
we're
focused
in
on
again
a
raw
input
stream
here,
which
is
an
ethernet
interface
and
I've
got
three
example,
policies
and
all
three
of
them
are
currently
tapping
into
this
eth0
and
when
they
do
that,
they're
able
to
specify
a
filter.
At
this
point-
and
in
this
case
it's
port
53-
and
this
is
filtering
at
the
kernel
level.
A
This
is
a
bpf
filter
and
it
means
all
of
these
policies
are
able
to
process
network
traffic,
which
is
focused
on
port
53.,
and
then
these
three
other
policies
that
we
see
here
are
working
concurrently,
and
so
the
first
policy
has
two
different
analyzers
attached
to
it.
One
is
a
network
analyzer
one
is
a
dns
analyzer
and
both
of
these
are
working
again
with
these
trimming
algorithms
to
pull
out
that
information,
and
so
we're
pulling
out
the
network
information
we're
pulling
out
the
dns
information.
A
But
we
also
in
this
first
policy,
have
added
another
filter
and
that's
where
we
are
focused
on
nx
domain
traffic
and
in
the
dns
world.
That
means
a
record
that
was
not.
There
was
queried,
and
so
this
first
policy
is
geared
towards
finding
the
queries
that
are
coming
in
that
are
causing
nx
domain
traffic
to
happen.
A
So
here's
a
console
view
on
the
left
hand,
side
you'll,
see
that
we're
querying
this
local
pac
advisor
instance,
which
is
running
on
a
port
here
and
it's
accessing
the
json
output
and
so
on
the
left
side.
You
start
to
see
things
like
the
rates,
the
top
queue
name,
two,
the
cardinality
of
q
names
that
we
discussed.
So
it's
all
available
in
a
generic
json
format
and,
on
the
right
hand,
side.
It's
querying
a
slightly
different
endpoint
and
you
see
the
prometheus
version
of
that
output.
A
We've
tried
to
make
pac
advisor
very
easy
to
use
and
install
so
it's
available
as
a
docker
container.
It's
also
available
as
a
static
binary,
but
in
this
instance
we're
seeing
that
we
can
just
pull
the
pac
advisor
if
we
want
to
start
with
a
very
simple
default
collection
policy,
here's
a
command
line
that
will
do
that.
We're
telling
it
that
we
want
to
run
the
packet
visor
daemon.
A
This
is
an
open
source
project,
it's
hosted
on
github
and
the
ns1
labs
github
organization.
We
do
welcome
contributors,
there's
there's
a
lot
of
different
ways.
You
could
contribute
to
the
project
and
we'd
certainly
love
to
see
that
please
start
the
project.
If
it's
interesting
to
you
as
well
lets
us
know
how
much
interest
we
have
in
the
project
and
let's
talk
about
pack
advantage
with
prometheus.
So
we
talked
about
pac
advisor
itself,
which
is
directly
on
a
node.
Now
we
want
to
get
a
more
global
view
than
a
single
node.
A
So
that
gives
us
this
global
view
and
we're
able
to
scrape
the
pac
advisor
agents,
as
we
saw
from
that
endpoint.
We're
also
able
to
use
remote
write,
which
I'll
show
in
just
a
second
and
once
we
do
that
all
those
metrics
that
we
saw
from
all
those
policies
now
go
into
the
prometheus
ecosystem
and
we're
able
to
use
familiar
tools
for
exploring
that
data
and
visualizing,
creating
dashboards
and
alerting.
A
That
looks
something
like
this,
so
we
can
imagine
that
these
pack
advisor
agents
are
installed
in
multiple
locations,
hybrid
topologies,
it's
it's
it's
a
sidecar
agent
style.
It
can
be
in
containers,
vm
servers,
we've
started
putting
this
directly
onto
routers
and
switches
and
again
the
idea
is
to
pull
this
information
into
prometheus
database
and
then
set
up
your
tools
to
get
this
broader
view.
A
There's
a
docker
container,
that's
available
that
helps
you
with
a
push-based
system,
so
pec
advisor
prom
right
is
available
on
the
ns1
labs
organization,
and
this
is
basically
pac
advisor.
That's
been
packaged
along
with
griffon
agent,
and
so
it
knows
how
to
locally
query.
The
pack
advisor
do
the
scrape
for
you
and
then
push
it
out
to
any
remote
write,
compatible
interface.
A
Here's
a
quick
little
movie
of
what
it
looks
like
to
see
some
of
those
metrics.
So
this
is
just
the
explorer
section
of
grafana.
If
we
type
dns
in,
we
start
to
see
that
list
of
metrics
that
packadvisor
is
generating
and
we
can
quickly
drill
down,
select
what
we'd
like
and
create
graphs
help
us
create
dashboards.
A
And
then
this
is
what
a
our
premade
dashboard
looks
like,
and
this
is
basically
all
the
information
we
saw
inside
of
the
pack
advisor
command
line
interface.
But
now
we've
pulled
out
to
that
that
larger
view
now
we're
able
to
select
different
nodes
from
different
regions
show
multiple
agents
in
the
same
dashboard
and
so
forth.
A
One
of
them
is
how
do
we?
How
do
we
organize
all
of
these
pac
advisors?
Now
we
have
the
ability
to
have
policies
that
are
spinning
up
and
down
inside
of
the
pac
advisors
in
real
time.
But
how
do
you
orchestrate
that?
How
do
you
actually
collect
and
access
these
agents
that
you've
installed
in
different
places
and
send
them
policies
and
collect
the
information?
A
A
A
It
provides
a
rest
api
to
be
able
to
do
automation
against
them
and
it
handles
agent
communication
to
talk
to
these
software
agents
and
the
whole
goal
here
is
being
able
to
orchestrate
the
agents
that
are
sorry,
the
policies
that
are
going
to
the
agents
and
to
be
able
to
collect
the
information.
That's
coming
back
for
them
as
they
spin
up
and
down
and
get
them
syncing
to
your
time.
Series
database.
A
A
So
it
lets
you
organize
your
policies
as
well,
be
able
to
send
them
out
in
real
time
and
then
it
handles
being
able
to
understand
which
policies
are
running
on
which
agents
collect
the
information
back
for
you
and
you
can
send
it
to
your
time
series
syncs!
It's!
It's
meant
to
be
used
in
two
ways,
and
so
there
is
a
helm
chart
available
that
will
help
you
deploy
to
kubernetes
and
you
can
run
the
control
plane
yourself.
A
We
will
also
plan
on
having
a
free
sas
if
you
don't
run
the
entire
control
plane
yourself
and
you'd
like
to
use
the
functionality
and
just
install
the
agent
on
your
devices.
We'll
have
we're
planning
sas
at
this
web
address
and
there
is
modular
support
for
the
observability
agent
too.
So
pac
advisor
is
essentially
a
module
for
orb.
We
might
even
run
other
types
of
observability
agents
inside
the
orb
agent
in
the
future
as
well.
A
Here's
a
quick
diagram
of
what
this
looks
like,
so
we've
got
the
pac
advisor
agents
coming
into
the
orb
control
plane.
It
uses
the
mqtt
protocol,
which
is
which
is
a
known
protocol.
That's
used
with
iot
and
we've
got
a
couple
of
micro
services
that
are
handling
the
control,
plane,
functionality
and
the
user
interface,
which
lets
you
organize
your
policies
and
your
fleet.
A
key
point
here
is
that
orb
itself
is
not
concerned
with
visualizing
the
actual
output
from
the
agents.
It
will
help
you
organize
things.
A
Quick
screenshot
of
what
the
user
interface
looks
like
this
is
under
heavy
development.
By
the
way
in
terms
of
where
orb
project
is,
our
goal
is
to
have
an
mvp
at
the
end
of
this
month
to,
for
the
first
time,
have
end
users
install
the
entire
control
plane
and
use
the
system.
So
again
it's
a
very
new
project,
but
here
we
can
see
that
there's
a
list
of
agents
that
have
that
have
logged
into
the
control
plane.
We
can
understand
when
the
agents
are
online.
There's
a
heartbeating
system.
There's
a
tagging
system.
A
And
that
about
winds
it
up
just
to
just
to
summarize
again,
this
is
pac
advisor
and
orb,
and
the
goal
is
to
to
fit
into
your
observability
stacks
and
to
help
you
with
edge
network
observability,
they're,
open
source
projects
and
the
entire
goal
here
is
dynamic,
orchestration
of
business
intelligence
at
the
edge
with
that
I
will
wind
it
up.
Thank
you
very
much
for
your
attention.
You
can
find
more
information
at
these
web
addresses.
A
My
email
address
is
here.
Please
contact
me
anytime.
If
any
of
this
is
interesting,
you'd
like
to
discuss
some
of
this
further,
we
do
have
an
announcement
list
here.
If
you're
interested
in
the
sas
please
join
there.
We
have
a
slack
as
well
and
I'll,
be
here
all
week.
If
anybody
would
like
me
to
demo
some
of
this
functionality,
I'm
happy
to
do
that
and,
of
course
you
can
contact
me
on
the
virtual
platform
as
well.