►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Monitoring Service Architecture and Health with BPF - Jonathan Perry, Flowmill, Inc.
Kubernetes has made it incredibly easy to build distributed applications out of large numbers of microservices. Monitoring, or even accurately tracking, the interaction between each of these services can be a significant operational challenge. In this talk, we will explain how you can get 100% visibility into the flow of data between services in your Kubernetes cluster using BPF (Berkeley Packet Filter). We will first discuss how flow data can help you monitor service architecture and assess the health of microservices. Next we will focus on how you can use BPF (extended Berkeley Packet Filter) to extract flow data without any changes to application code, aggregate it across pods, services, and namespaces, and analyze it with Prometheus. Finally, we will show this in action in our production cluster, and discuss some of challenges that emerge as you roll it out at scale.
https://sched.co/MPbs
A
Thanks
everybody
I'm
excited
to
be
here.
Let's
dive
right
in
in
this
talk,
I
want
to
discuss
how
you
can
get
visibility
into
your
clusters,
architecture,
health
and
cost
using
BPF
and
I
want
this
talk
to
me
top-down,
so
I'm
gonna
start
with
a
demo
and
show
you
what
kind
of
visibility
you
would
want
to
get
then
go
into
how
you
could
collect
this
type
of
telemetry,
spoiler
alert.
A
A
quick
note
about
myself.
I'll
tell
you
a
bit
about
my
history.
I
first
was
exposed
to
these
large-scale
operational
challenges
as
part
of
work.
In
a
government
technology
agency,
I
spent
seven
years
there
I
started
by
developing
systems
that
were
deployed
at
scale
in
the
field
mission
critical
systems,
but
it
quickly
became
apparent
that
it
that
we
have
an
Operations
problem.
A
We
had
10
to
20
operators
who
kept
fire
fighting
waking
up
in
the
middle
of
the
night
to
deploy
configuration,
changes,
restore
connectivity
and
deploy
new
software
versions,
so
I
switched
to
the
infrastructure
organization
and
built
systems
to
monitor
and
automatically
and
semi-automatically
mitigate
failure.
I
then
moved
to
MIT
to
do
a
PhD
in
these
monitoring
and
mitigation
systems
that
are
extremely
low
overhead,
low
latency
high
throughput.
For
example,
we
had
a
system
called
Fast
Pass
where
there
was
a
centralized
server,
which
we
called
the
arbiter.
A
A
So,
according
to
organizational
policy,
so
this
arbiter
allocated
2
terabytes
of
traffic
of
data
of
traffic
per
second
with
50
microseconds
of
latency,
with
deployed
at
Facebook
in
production
during
2013.
So
this
is
just
to
give
you
a
background
into
what
technology
were
bringing
into
this
type
of
visibility.
I
started
flow
mill
two
years
ago
in
order
to
get
better
visibility
for
operations,
so
I'm
gonna
start
with
this
demo
application
I.
Think
many
of
you
know
this.
It's
a
great
it's
a
great
open
source
project.
A
I
want
to
thank
Google
platform
for
open
sourcing
this,
and
this
is
the
hipster
shop.
So
if
you
haven't
seen
it
it's
this
ecommerce
site
that
you
can
deploy
in
communities,
you
can
go
and
buy
these
hipster
items
maybe
buy
four
of
these
terrariums.
It
has
a
shopping,
cart,
checkout
I!
Could
you
can
place
your
order
and
such
so?
How
does
this
system
look
in
practice?
Let's
go
in
and
see
how
the
deployment
looks.
So,
usually
you
have
this
kubernetes
manifest.
That
tells
you!
What
are
the
services
you
can
see
here?
A
There's
a
front-end
service,
the
cart
service,
the
checkout
service
and
so
on.
Usually,
when
you
want
to
operate
these
systems,
this
is
not
enough.
What
you
want
is
an
image
of
how
how
your
system
interacts,
how
your
components
interact
and
how
your
system
is
built.
So
luckily
many
organizations
create
these.
These
architecture
diagrams
and
the
hipster
shop
also
has
this
architecture
diagram.
A
Unfortunately,
these
architecture
diagrams
are
frequently
not
up-to-date.
So
when
you're
dealing
with
an
incident,
it
gets
really
hard
to
understand
what's
happening.
What
you
would
want
is
something
like
this,
where
you
reconstruct
the
architecture.
If
your
system
from
live
data
in
real
time
here,
you
can
see
the
front-end
service,
the
load
generator
and
so
on.
A
So
if,
if
we
go
back
to
the
architecture
diagram,
what
I
did
is
I
rearranged
all
the
services
in
the
real
time
graph,
and
so
that
we
can
overlay
the
architecture,
diagram,
I,
hope
this
is
visible
and
what
you
can
see,
that
is
that
with
the
vanilla
deployment
of
the
hipster
shop,
the
email
service
and
the
payment
service
are
not
in
use.
So
these
are
the
type
of
things
that
you
really
want
when
you
operate
services
at
scale.
A
The
other
thing
you
can
get
out
of
this
when
you
have
real
time
data
is
when
you
have
a
deployment
you
make
you
deploy
a
new
version
of
a
service
you
want
to
understand,
did
the
did?
Does
it
have
the
same
dependencies?
Is
it
getting
it?
Does
it
have
a
new
dependency?
Is
the
old
dependent
are
the
old
dependencies
all
working
as
you
expect,
so
this
gives
you
confidence
when
you
deploy
another
thing
real
time
architecture
map
gives
you
is
the
ability
to
understand
your
load,
balancing
and
high
availability
architecture.
A
A
A
A
Okay,
second
item,
so
we
went
through
architecture
now,
I
want
to
discuss
health.
How
can
you
find
what's
happening
in
your
cluster,
using
this
data,
so
I'm
going
to
introduce
service
degradation
in
this
product,
catalog
service
I'm
going
to
do
some
manual,
chaos
engineering?
So
let's
go
to
these
these
nodes.
A
A
Sometimes
you
can
have
a
system
do
this,
for
you,
I'm
gonna
do
this
manually
for
the
demo,
so
this
is
interface,
100,
142
I'm,
then
gonna
SSH
to
the
kubernetes
node
and
get
a
list
of
IP
addresses
so
where
we
want
142
here
so
here
it
is
so.
This
is
the
network
interface
on
the
node
that
is
matched
to
the
pod
interface,
so
142
so
I'm
gonna.
Add
some
delay.
A
Let's
say
75
milliseconds
before
I:
do
this
I
want
to
introduce
kind
of
the
dashboard
you
can?
You
can
use
to
look
at
this?
So
here
is
the
same
data
that
you
saw
in
the
graph
the
map
view
in
dashboards.
So
you
can
see
TCP
throughput,
drops
and
round-trip
time.
Also
CIN
timeouts
I'll
talk
about
these
of
it
later
and
here,
I've
filtered
to
only
show
the
product
catalog
service.
I
know
this
must
be
a
bit.
The
font
might
be
small,
but
I.
Don't
think
you
need
to
read
it
right
now.
A
A
Remove
this
delay
so
yeah
the
system
should
recover
very
quickly
to
remove
this.
So
the
round-trip
time
here
decreases
here
you
have
a
sample
that
has
low
round-trip
time.
So
this
is
one
of
the
one
of
the
types
of
servers,
the
gradations
that
you
want
your
system
to
show
you
of
course,
I'm
showing
you
dashboards,
but
of
course
you
can
create
alerts
based
on
this.
The
other
thing
I
want
to
show
you
is
maybe
discovering
if
your
security
groups
are
misconfigured,
so
I'm
going
to
deny
traffic
going
to
the
Rena
server.
A
The
Redis
node
here
I'm,
going
to
do
this
by
just
editing
manually,
editing
the
the
security
group,
usually
you'd
have
a
typo
in
your
terraform
or
whatever
mark.
Thank
you,
whatever
technology
you're
using.
So
this
is
the
port
six,
three
seven
nine,
but
usually
used.
Let's
say:
I,
not
fingered
it
and
wrote
a
zero
here.
Let
me
introduce
the
dashboard
first,
so
this
is.
The
kind
of
this
is
the
similar
dashboard,
where
I
filtered
only
to
the
Redis
node.
A
Refresh
and
very
quickly,
we
see
sin
timeouts,
so
these
are
timeouts
in
connection
establishments
and
so
also,
if
you
could
alert
on
these,
so
these
shouldn't
really
happen
and
there
definitely
shouldn't
happen
like
multiple
times
in
consecutive
seconds.
So
this
is
another
type
of
a
fault
that
you
can
discover
with
flow
data.
I'm
gonna,
just
River.
This.
A
Okay,
I'm
breathing
through
this
I'm,
sorry,
but
let's
head
to
the
third
category,
which
is
cost
so
we
don't.
We
did
architecture
and
health
and
now
cost.
When
you
run
on
the
public
cloud,
there
is
a
dollar
cost
for
communication
between
availability
zones
and
an
even
higher
cost
for
communication
between
regions.
A
Can
you
get
a
real-time
view
of
how
your
application
is
behaving,
so
you
can
optimize
it
so
here
here's
a
dashboard
where
I
filtered
only
through
the
traffic
from
this
demo,
I'm
gonna
zoom
in
on
this
throughput
graph,
and
what
I'm
gonna
do
now
is
I'm
gonna,
add
a
breakdown
of
source
traffic
by
availability
zone.
This
adds
this
column,
so
you
can
see
traffic
going
from
the
front
end
on
to
be
to
any
node
of
low
generator.
A
Regarding,
regardless
of
what
every
lability
zone
is
in
now,
I
can
add
to
breakdown
on
this
on
the
destination
side.
So
now
you
can
see
traffic
from
on
the
front
end
from
to
be
to
the
load
generator
to
a
to
be
to
to
a
and
so
on
and
I'm
gonna.
Add
another
filter.
You
can
sorry
I'm
gonna
filter
only
traffic
between
different
zones.
So
usually
you
do.
It.
You'd
have
a
dashboard
that
does
this,
but
I
wanted
to
show
the
filtering
so
here
sort
by
total
amount
of
traffic.
A
A
So,
just
to
summarize
so
far,
we've
seen
these
five
use
cases
across
three
categories
in
the
architecture:
getting
more
confidence
in
your
deployments
and
having
a
real-time
view
of
architecture,
understanding
your
high
availability
architecture
and
load-balancing.
By
the
way.
This
is,
if
there's
one
slide,
you
remember
out
of
this
talk.
This
should
be
it
right.
So
for
health.
You
can
understand,
you
can
see
service
the
gradation
very
quickly
and
understand
security
groups
and
for
cost
you
can
see
traffic
across
zones
or
regions,
and
you
can
do
all
of
this
with
flow
data.
A
Ok,
now
we
did
this
I
want
to
go
into
how
you
get
this
flow
data.
What
is
in
this
data?
You
assemble
it
from
two
sources
from
the
Linux
kernel.
You
can
get
information
on
very
frequently,
let's
say
every
second
on
what
each
source
is
talking
to
for
every
socket
right.
What
each
source
IP
address
destination,
IP
address
and
ports
are
talking
of
how
much
they're
sending
the
number
of
packet
drops
round-trip
time
and
sync
timeouts
and
such
from
kubernetes.
A
You
can
get
metadata
of
the
pod,
so
the
IP
address
the
name
of
the
deployment
and
the
pod
of
the
image
that
is
serving
the
the
for
each
container,
the
tag
of
each
image.
So
this
is
the
version
of
your
service
running
and
the
zone
that
it's
running
in
so
all
of
these
are
failure,
domains
that
you
can
then
extract
telemetry
for
and
understand,
if
that
is
that
failure
domain
is
affecting
your
service.
A
A
I'm
gonna
go
through
so
I've
told
you
what
you
expect
to
get
out
of
it,
how
to
get
it
and
then
what
what
are
the
benefits
and
the
disadvantages?
So
the
advantages
are
one.
You
don't
need
to
change
your
application,
that's
great,
unlike
with
custom,
metrics
or
logs
or
traces,
where
developers
constantly
have
to
maintain
the
instrumentation
in
the
application.
Here
you
get
everything
from
the
kernel
and
Orchestrator,
so
there's
the
deployment
is
easier
in
those
lower
organizational
cost.
A
Unlike
other
technologies,
here
you
don't,
we
don't
have
to
support
many
languages
and
many
frameworks.
So
it's
very
easy
to
hyper
optimize
all
the
collection
points,
so
you
can
get
very
low
overhead
collection
and
there's
more
about
this
in
the
evaluation
section,
I'll
show
you
some
benchmarks
and
results
and
you
can
get
visibility
into
external
services,
external
api's
managed
services,
databases
and
so
that's
the
same
way
as
you
get
for
your
normal
services.
A
The
disadvantage.
There's
one
major
major
disadvantage:
the
the
flow
flow
data,
as
I've
shown
you
so
far,
doesn't
collect
application,
layer,
error,
codes
or
metrics
at
all,
and
usually,
when
kind
of,
when
you
operate
for
a
business
purpose,
you
want
to
understand
how
the
users
are
experiencing
the
application.
What's
failing
and
what's
not-
and
this
doesn't
do
that
it
has
these
advantages,
but
it
doesn't
collect
the
error
codes
you
can
get
proxies.
A
So
usually
we
found
that
throughput
of
a
service
is
a
proxy
for
service
health,
because
once
the
service
starts
sending
400
or
500
error
codes,
the
error
responses
are
tiny
compared
to
regular
payloads.
So
you
immediately
see
a
drop
in
throughput,
so
you
can
use
that,
but
still
it's
a
proxy.
This
is
something
that
we
want
to
fix
and
I'm
gonna
I'm
going
to
discuss
this
very
shortly.
At
the
end
of
the
talk
towards
the
end
of
the
talk,
how
you
could
solve
that
I
wanted
to
front-load
that
a
disadvantage,
but
the
advantages
are
large.
A
A
With
a
virtual
IP
address,
Forex
a
establishes,
a
connection
to
X
and
then
iptables
maps
that
X
address
to
B's
address
so
map's,
the
a
to
X
connection
to
a
to
b
and
b
just
sees
packets
as
if
they're
coming
directly
from
a.
So.
How
do
you
get
this
flow
data?
First,
all
the
Padma.
Today
you
can
get
from
the
kubernetes
api,
the
socket
level
information
you
can
get
with
the
SS
utility
and
linux.
So
this
is
widely
available
here.
A
You
can
see
a
2x
communicates
how
many
bytes
there's
also
run
trip
time
and
other
metrics
and
from
contract.
You
get
information
about
the
IP
tables,
the
actual
rose
in
the
NAP
table.
So
here
you
see
the
mapping
from
a
to
X
to
A
to
B.
So
this
is.
This
is
great.
Like
all
the
tools
are
there,
you
can
build.
A
solution
like
this
cui
has
a
problem,
which
is
what
is
going
to
lead
me
to
BPF.
First
is
performance
that
the
performances
is
actually
not
the
deal
breaker
but
I
want
to
address
it
first.
A
So
here
is
a
test
that
we
ran
on
a
production
machine.
It
shows
on
the
x-axis,
the
number
of
sockets
and
on
the
y-axis
how
many
million
cycles
it
took
to
run
SS.
You
can
see
that
kind
of
from
this
results
of
the
regression
line,
every
socket
that
that
SS
touches
cost
79
thousand
cycles
and
when
you
run
this
every
second-
and
you
have
a
lot
of
these
sockets
open
on
the
machine,
then
it's
gonna
get
expensive
because,
and
the
reason
is
the
tool
is
built
for
the
CLI.
A
Another
reason:
is
it
a
great
so
ver
all
the
sockets
in
the
system,
even
if
they
didn't
have
any
data
on
them,
so
you're,
paying
even
for
sockets
that
are
not
helpful
to
you,
but
maybe
the
you
know.
More
important
problem
is
coverage.
The
CLI
tools
are
built
for
polling,
so
what
you
do
is
maybe
every
second
or
every
10
seconds.
You
run
the
tool
and
get
a
snapshot
of
the
system
state,
and
this
tends
to
miss
a
lot
of
events.
A
A
What
it
allows
you
to
do
is
it
allows
user
space
programs
to
specify
code
snippets
that
the
kernel
then
runs
on
behalf
of
the
user
space
programs
on
different
kernel
events,
and
usually
these
are
functioning
executions
in
the
kernel.
So
when
different
things
happen
inside
the
kernel,
so
the
snippet
of
code
runs,
it
allows
you
to
space
programs
to
extract
data
from
the
kernel
very
efficiently.
A
One
advantage
is,
you
can
get
only
changes
into
intestate
in
the
kernel,
so
only
sockets
that
actually
had
data
on
them,
rather
than
all
the
sockets,
for
example,
and
it
gives
you
access
to
internal
data
structures
where
you
can
extract
more
data
that
is
not
usually
easily
available
to
user
space
through
system
calls.
The
gif
is
very
safe.
It's
built
with
verifier
in
the
kernel
that
ensures
that
vpf
code
doesn't
access
invalid
memory,
doesn't
loop
forever
and
so
on.
A
In
fact,
loops
are
forbidden,
beefy
at
EPF
code
completely
and
the
in
the
contexts
in
all
the
contexts
where
we
kind
of
this
talk
and
runs
BPF
code
in
all
those
contexts.
The
memory
and
the
kernel
is
read-only,
so
you
cannot
change
anything
in
the
kernel.
Bpf
is
also
very
fast.
The
kernel
contains
a
just-in-time
compiler
that
takes
these
byte
code
snippets
and
compiles
them
into
very
efficient
machine
code.
So
you
can
get
this
100%
coverage
with
very
low
overhead.
A
I
want
to
run
through
a
quick
demo
of
how
BPF
utilities
of
some
VB
fit
utilities
I'm
going
to
run
on
one
of
the
kubernetes
cluster.
Is
this
utility
called
TCP
top
from
the
BCC
repository?
It's
a
great
open
source
project
that
Python
bindings
and
many
utilities
accessing
kernel
data
structures
with
you,
Kiev.
A
A
Sorry,
I'm
gonna
run
it
again
just
so
we
can
see
it.
So
you
can
see
here's
the
SSH
that
we're
using.
In
order
to
view
this
data,
you
can
see
the
different
services
in
the
hipster,
shopped,
communicating,
here's,
the
cart
service
and
so
on
and
by
instrumenting
these
kernel
a
paternal
functions
so
specifically
tcp
top
instruments
sent
message
and
tcp
cleanup
are
buff.
A
This
items
call
actually
goes
and
fetches
data
from
the
kernel
data
structures
that
are
shared
between
BPF
code
and
user
space,
then
analyzes
them
does
something
it
puts
them
into
a
dictionary
and
then
clears
that
data
structure
and
kind
of
the
the
reason
I'm
showing
this
is
because,
when
you
write
this
type
of
code,
you
need
to
be
careful
about
types
of
races.
For
example,
in
this
case
between
the
time
the
the
user
space
program
reads
all
the
state
from
the
kernel
and
the
time
it
clears
the
data
structure.
A
So
it
doesn't
have
to
read
the
same
values:
the
kernel
can
insert
new
sockets
and
all
those
sockets
are
cleared
here
thrown
away.
So
these
are
the
types
of
things
you
need
to
be
careful
when
you
build
a
system
for
real.
This
would
give
you
by
the
way
the
race
here
is
pretty
small,
so
it
will
give
you
99%
coverage
right,
I
know,
depending
so
you'd
miss
1%
of
traffic,
but
maybe
you
don't
want
to
miss
that
like
we,
when
we
build
these
systems,
we
take
care
of
all
the
edge
conditions.
A
We
don't
want
to
have
this,
so
we've
gone
through
BPF
and
kind
of
demo
and
kind
of
what
things
to
pay
attention
to,
and
I
want
to
talk
a
little
bit
about
how
to
build
this
end-to-end
as
as
a
system
to
analyze
this
code.
So
first
and
the
agent
interact
with
the
kubernetes
api
and
also
other
orchestrators,
could
be
yes,
docker,
ec2
and
extracts
data
from
linux.
It
then
forwards
data
into
an
analysis
pipeline.
A
The
goal
of
the
analysis
server
here
is:
first,
if
you
have
an
agent
on
both
sides
of
the
connection,
then
your
agents
on
both
sides
of
the
connection
you
want
to
match
the
two
flows
so
that
you
get
a
hear
interview
of
that
flow
from
both
sides.
It
then
enriches
that
flow
with
metadata
from
the
orchestrators
and
aggregates
them.
A
We
use
Prometheus
right
now
and
we
found
that
you
can
build
very
cool
api's
on
top
of
this,
because,
usually
when
you
have
time
series
databases,
the
labels,
are
you
have
one
set
of
labels
here
where
the
this
flow
data,
you
have
a
set
of
labels
for
the
source,
the
set
of
data
for
the
destination
and
a
set
of
labels
for
the
flow
itself.
So
the
api's
could
be
much
more
convenient
you
can.
You
could
build
api's
that
are
more
convenient
to
work
with
this
data.
A
What
you'd
want
more
is
to
build
an
analysis
pipeline
so,
for
example,
I
to
analyze
this
data
and
create
events.
So
one
is
like
the
standard
alerting
that
you
get
from,
for
example,
graph.
No
other
types
of
exact
exciting
you
can
build
our,
for
example,
this
version
detector,
where
I
have
a
new
deployment,
I
detect
a
new
version
and
I
want
to
see
our
is
the
new.
Does
the
new
version
behave
the
same
as
the
old
version,
so
these
are
pretty
exciting
things
you
can
do
and
we're
looking
for
feedback
there.
A
So
if
you
have
cool
ideas
or
want
to
discuss,
I
really
love
to
talk
about
that
more
okay.
So,
let's
jump
into
evaluation
of
the
system.
So
how
does
it
behave?
Is
it
gonna
be
too
expensive
to
run?
We've
run,
perform
measurements,
I've
included
here
kind
of
our
methodology.
Perf
is
a
sampling
profiler,
so
you
get
these
stacks
execution
stacks
from
the
kernel
every
here,
5
million
cycles
and
you
can
analyze
those
and
see
how
many
cycles
are
statistically
in
each
in
each
function.
A
One
thing
to
note
is:
when
you
evaluate
these
monitoring
systems,
you
need
to
evaluate
not
only
the
user
space
portion,
which
is
what
is
readily
accessible,
but
also
the
kernel
overhead.
So
all
of
the
measurements
I'm
going
to
show
I
mean
there's
not
a
lot,
but
we
have
we
evaluate
the
criminal
overhead
as
well,
which
could
be
more
it's
frequently
more
than
the
user
space
part.
A
What
we've
seen
is
the
collector
that
we've
been
using
has
point
one
to
point:
two:
a
quarter
percent,
pork
or
CPU
overhead.
Across
many
deployments
we
do
keep
track
of
the
highest
overhead
we've
ever
seen.
We've
had
a
one
of
our
customer
have
run
a
very
aggressive
load
test
on
their
servers
that
their
application
worked
at
46%,
which
is
very
high
for
them,
and
with
that
application
our
collector
ran
with
less
than
a
percent
0.86%
CPU
overhead.
So
this
is
the
highest
that
we've
seen.
A
Is
this
gonna
be
expensive
in
network
overhead
now
you're
sending
all
the
socket
telemetry
every
second
to
the
analysis
pipeline?
So
here
we
have
several
different
clusters
that
are
running.
These
are
production
clusters
running.
This
is
megabytes
per
second.
In
throughput
on
the
cluster
and
the
workload
itself-
and
you
know,
this
is
a
flow
data
collection
service,
so
it
also
monitors
the
amount
of
telemetry
flowing
to
the
flow
to
the
flow
analysis
pipeline.
A
So
here's
the
amount
of
telemetry
and
megabytes
per
second,
and
we
found
that
in
most
most
cases
you
can
get
under
0.5%
CPU
overhead
and
if
you're
running
a
very
intense
workload
where
you
have
these
tiny
tiny
transactions
and
tons
of
them,
then
you
get
higher
network
overhead.
This
is
Wow.
1.15
is
what
we've
seen.
Outliers
are
yeah
around
1%
another
point
on
how
to
build
these
analysis
pipelines
what
you
could
expect.
A
The
number
of
events
here,
I've
I've
shown
three
different
companies
deployment
and
the
number
of
events
that
you
get
of
every
different
types
per
second
per
agent.
So,
for
example,
company
a
has
1,400
TCP
events
per
second
per
agent
here
might
be
where
you
want
to
look
like
total
number
of
events
per
second
for
agent.
So
you
could
expect,
if
you're
running
a
batch
workload
to
have
around
100
events
per
second,
if
you're
running
a
more
intense
interactive
workload
than
several
hundreds
to
thousands.
A
A
This
is
partly
why
we
were
motivated
to
write
this
in
C++
to
make
sure
we
can
control
the
overhead
and
latency
and
we
were
able
to
support
several
hundred
nodes
with
a
two-second
latency
target
and
hope
to
have
thousands.
So
this
is
achievable.
You
just
need
to
pay
attention
to
the
details,
so
this
so
here,
gonna
we've
done
evaluation.
I
know
this
is
a
very
fast-paced
talk,
so
this
just
summarizes
the
evaluation
and,
at
this
point
I
want
to
touch
on
the
disadvantage
that
we
that
I
mentioned
earlier,
getting
application-layer
metrics
or
error
codes.
A
A
These
are
all
the
symbols
that
it
has
all
the
applications
and
Allah
ID
and
BPF
allows
you
to
hook
into
these
functions
and
get
telemetry.
We
let
me
run
it
so.
I'm
gonna
run
the
hello
world
server
then
use
the
BPF
utility
funk
on
this
PID
and
what
I'm
gonna
do
is
instrument
root,
hello
and
get.
This
is
a
go
a.
A
A
Okay
and
then
I'm
gonna
curl
the
the
whole
world
server
and
get
some
responses,
so
here,
I'm
gonna
do
it
three
times
and,
as
you
close
the
the
funk
count
tool,
you
can
see
that
BPF
was
able
to
get
to
run
three
times
on
this
function.
Right
header
and
specifically
the
right
function
right
header.
This
function
is
where
the
error
code
goes
out,
of
the
go
length
and
at
HTTP
handler.
So
it
is
very
possible
to
write
these
BPF
probes
that
get
application
level
errors.
A
So,
with
this
I've
we've
talked
about
flow
monitoring:
how
to
get
architecture,
health
and
cost
visibility.
The
benefits
are
very
easy
deployment.
You
don't
have
to
change
it.
The
code,
low
overhead
and
visibility
into
external
services-
hopefully
application
metrics
hope
to
in
the
next
talk
to
talk
about
that.
More
and
BPF
is
great
technology
for
that,
and
with
that
I'd
love
to
take
questions
and
please
reach
out
later
we'd
love
feedback,
we're
in
kind
of
this
early
access
mode.
It's
very
exciting,
we'd
love
to
talk
to
the
community.
Thank
you.
B
B
A
You're
asking
if
you
can
use
BPF
in
order
to
interject
into
your
code
and
modify
it
well.
Vpf
is
specifically
a
lot
of
it
is
written
as
a
read-only
system.
So
if
you
want
to
write
tests
with
it,
it
gets
a
little
bit
hard.
There
are
places
in
the
kernel
where
you
can
modify
packet
contents,
for
example,
but
I
think
it
might
be
heavyweight
for
use
case.
If
you
work
a
lot
with
packets,
you
might
be
able
to
use
a
VP
of
that
way.
C
Hi
I'm,
thanks
for
the
impressive
tool.
I
have
two
questions,
so
it
wasn't
clear
to
me
like
whether
you
are
collecting
data
from
the
host
OS
of
a
node
or
the
OS
for
in
inside
docker
images
and
the
second
question
I
have.
Is
it
feels
like
the
correlation
between
the
OS
data
and
the
kubernetes
data
might
not
be
100%
correct,
especially
when
there
are
dynamic
changes
to
iptables
I,
don't
know
like
pod
going
up
and
down
and
state
changes
in
kubernetes.
A
Okay,
so
the
first
question
was:
are
we
monitoring
inside
containers
or
outside
containers?
Well,
containers
are
built
as
a
lightweight
isolation
mechanism
in
the
Linux
kernel,
so
by
running
as
a
privilege,
so
you
can
run
as
a
privileged
container
or
on
the
host
and
see
everything
that's
running
on
the
host.
So
you
get
this
visibility
into
the
containers
while
running
one
agent.
So
this
is
not
a
sidecar.
You
can
run
just
one
privilege
daemon
set
on
your
machines.
Your
second
question
was
I'm.
A
So
how
do
you
match
the
data
between
your
Orchestrator
and
your
notes
that
the
correctness,
the
correctness
is
correct,
yeah
yeah
so
luckily
many
of,
for
example,
containers,
have
a
unique
ID
that
is
shared
across
the
operating
system
and
kubernetes.
So
if
you
look
at
your
kubernetes
pod
description,
when
you
see
the
containers
in
there,
it
has
the
same
UID
that
is
used
inside
kernel
data
structures,
so
you
can
match
those
and
we
do
so
kind
of
the
the
system
we
built
it
does
with
pids.
For
example,
you
can
have
these
problems.
A
If
you
collected
process
identifier,
then
you'd
have
this
problem
where
maybe
the
kernel
reuses
the
same
PID
multiple
times,
but
you
rarely
need
to
use
that
TCP,
for
example,
so
port
numbers
are
not
reused
very
quickly,
usually
so
you
could
rely
on
that,
but
frequently
we
just
rely
on
container
information,
so
you
know
the
socket
what
container
it
was
and
then
the
other
side.
What
container
the
saw
the
other
side
of
the
socket
was.
So
you
can
match
the
metadata
that
way,
and
that's
there's
very
little
error.
You
can
do
that.