►
From YouTube: TGI Kubernetes 154: Pixie
Description
Come hang out with Evan Anderson as he takes a look at Pixie, a new monitoring project being donated by New Relic. Let's dig in and figure out how Pixie relates to Prometheus and other monitoring technologies!
A
A
Happy
friday,
it's
sunny
out
I'm
in
here
anyway,
but
probably
after
this
I'll
be
going
out
and
enjoying
the
sun.
A
A
A
Seattle,
seattle
has
gotten
a
nice
burst
of
weather.
Here
it's
been
sunny
all
week.
A
I
bet
we'll
figure
out
why
a
new
monitoring
project
too
I've
only
looked
at
it
a
little
bit,
but
I
think
there's
some
interesting
stuff.
So
let's
see
we're
going
to
switch
over
to
switch
over
to
the
notes
and
do
a
little
week
in
review
to
start
with.
A
So
for
those
of
you
who
are
new
here
or
haven't
done
this
before
these
notes
are
at
tgik
dot,
io,
slash
notes,
and
I
will
see
if
I
can
put
that
in
the
chat.
Yes,
youtube
doesn't
like
links
in
there.
So
if
you
try
to
put
in
a
link,
it
may
get
broken
unless
you
break
them
up
so
yeah.
A
Anything
cloud
cloud
native
related
happened
last
week,
yeah
kubecon,
so
I
pulled
out
one
talk
that
was
interesting
to
me,
but
if
other
people
want
to
call
out
talks,
the
notes
are
open
in
hackmd.
B
A
Two
two
other
things
I
thought
were
interesting:
there's
a
release,
121
webinar,
which
is
going
to
be
describing
all
the
features
in
121
from
the
folks
who
actually
ran
the
release
and
that's
going
on
later
this
week.
So
if
you're
curious-
or
I
guess
next
week-
it's
friday
and
there's
a
cap
for,
swap
that
I
thought
that
some
people
might
be
interested
in.
A
Somebody's
got
the
video
playlist
in
here.
Oh
yes,
so,
the
last
time
I
looked,
they
hadn't
gotten
it
all
into
youtube.
Yet
so
yeah
it
looks
like
here
is
the
link
to
all
the
youtube
content:
we're
not
gonna
yeah.
You
know
how
it
says,
turn
off
swap
well,
who
knows
in
a
release
or
two
we
we
might
end
up
saying
actually
swap
can
be
on.
A
Swap
like
paging
out
to
disk.
A
Thanks
rory
yeah
kublet
cubelet
will
refuse
to
start
up
if
swap
is
on
because
it
turns
out.
If
you
move
random
pages
from
memory
to
disk,
your
performance
gets
really
unpredictable,
and
so
at
google
swap
has
been
off
for
years
and
years
and
when
kubernetes
started
up,
they
did
the
same
thing,
but
it
looks
like
some
workloads
really
benefit
from
swaps.
So
oh
awesome,
the
kubernetes
podcast
was
here
too
and
so
yeah.
A
I
thought
it
was
interesting
that
we're
going
to
be
getting
swap
on
kubernetes
and
yeah.
Let's
see
what
else
so,
yeah
kubecon
there's
the
video
list.
This
is
an
interesting
project
that
I
saw
on
twitter
and
joe
called
out
as
well.
So
kcp
is
a
prototype
of
kubernetes
without
pods
or
nodes
or
any
of
the.
I
want
to
run
code.
It's
just
an
api
server
for
those
of
you
who
might
want
to
be
able
to
write
kubernetes-like
apis,
but
don't
actually
need
to
do
pods
or
nodes.
A
A
Martin,
can
you
stick
that
in
the
in
the
hack
md
the
hacking
into
kubernetes
security?
That
seems
pretty
interesting.
A
And
then
the
it
looks
like
the
cncftoc
accepted
a
whole
bunch
of
new
projects,
so
I
figured
I'd
call
them
all
out.
Andrea
is
a
cni
that
uses
open.
What
is
it
uses
some
of
the
software
to
find
networking
stuff
and
I'm
trying
to
remember
what
is
it
called
open
v-switch,
so
it
uses
packet
level
forwarding
inside
of
the
node,
rather
than
the
some
of
the
bgp
or
other
custom
routing
protocols,
so
seemed
kind
of
interesting
chaos.
A
Blade
is
a
chaos
engineering
toolkit
so
lets
you
take
apart
your
cluster
in
random
ways
and
see
if
it
still
works.
A
So
fluid
is
a
tool
for
managing
large
data
sets
to
go
along
with
like
machine
learning
and
ai,
and
so
it
gives
you
a
bunch
of
crds
for
describing
where
your
data
is
across
different
storage
areas.
A
So
submariner
lets
you
connect.
Oh
it'd,
be
really
nice
if
they
had
a
picture
here
it
lets
you
connect
multiple
kubernetes
clusters
together
without
needing
to
go
through
exit
from
the
cluster
and
then
ingress
through,
like
an
ingress
or
a
gateway
somewhere
else,
and
they
can
kind
of
pipe.
You
know
pipeline
stuff
through
that
channel.
A
Vineyard,
I
didn't
look
too
deeply
into
this,
but
it's
a
mechanism
for
whether
you're
building
on
a
single
machine
or
on
many
machines
having
sort
of
a
distributed
file
system
and
efficiently
copying
things
between
your
preprocessor
and
your
next
data
stage
pipeline.
A
I
was
just
going
to
put
everything
under
the
video
playlist
heading
or
take
it
out
and
let's
see
the
last
one
was
wasm
edge,
which
is
not
the
sort
of
thing
that's
going
to
make
for
a
flashy
demo,
but
is
a
webassembly
virtual
machine
so
that
you
can
easily
use
webassembly
in
a
bunch
of
different
environments,
and
so
yeah
they've
got
how
to
run
it,
but
they
don't
show
you
how
to
run
doom
on
it.
So
I
can't
really
show
you
anything
exciting.
A
A
A
A
They
did
sort
of
here's
how
to
do
the
same
response
engine.
So
when
you
see
certain
unexpected
system
calls,
you
can
react
to
it
and,
for
example,
shut
down
the
pod
and
they
did
a
series
of
blog
posts
with
in
conjunction
with
different
projects.
So
three
serverless
platforms
and
then
techton
as
well.
A
A
B
A
Someone
saw
this
and
said:
hey,
I'm
going
to
show
you
how
you
do
it
with
tekton,
but
yeah
so
falco
has
built,
has
built
a
integration
using
cloud
events.
So,
instead
of
needing
to
tie
in
really
hard
with
one
of
these
particular
systems,
you
can
take
those
cloud
events
and
run
them
anywhere.
B
A
Let's
see
unless
anyone
else
has
billing
every
time
your
pots
on
a
scale.
Yeah
looks
like
there's
nothing
else
on
these.
You
know
in
the
comments
so
we're
gonna
start
looking
at
pixie,
lab
or
pixie,
which
it
looks
like
the
commercial
version
is
branded
this
way,
and
then
they
have
an
open
source
version.
A
That
looks
basically
the
same,
but
this
one
is
technically
the
new
relics,
the
new
relic
one,
and
this
is
where
pixie
labs
is
gonna,
be
so,
let's
see
so
they
have
just
do
this
and
it'll
all
work.
I
think
well,
there's
a
there's
a
note
that
there's
a
self-hosted
install
script
coming
soon.
So
now,
let's
someone
was
asking
why
a
new
monitoring
project?
A
Let's
do
that
one
first,
so
I
don't
know
how
many
people
have
used
new
relic
before,
but
they've
been
in
this
application
monitoring
space
for
a
long
time-
and
you
can
kind
of
see
from
this
little
thing
over
here-
they've
got
an
ability
to
sort
of
track
all
the
calls
between
your
different
components
and
build
you
traces
and
sort
of
why
things
are
slow
and
it
looks
like
pixi
also
has
the
ability
to
kind
of
have
a
script,
and
you
know
get
data
out
of
the
database,
and
here
is
how
it
all
works,
so
they've
got
modules
on
each
node
and
possibly
also
off
node,
and
then
a
central
control
server
that
talks
to
a
potentially
hosted
api.
A
So
inside
your
cluster,
you
have
these
edge
modules
on
each
node,
lots
of
different
systems,
sort
of
have
that,
but
they
hook
in
with
ebpf
so
that
they
can
get
kernel
level
syscalls,
and
then
they
do
a
kubernetes
native
collection
or
kubernetes
native
system.
With
this
vizier
agent
that
collects
all
the
data
from
the
different
pixy
edge
nodes,
and
then
they
have
an
option
to
send
things
to
their
cloud
or
do
a
direct
connection
called
a
called
a
called
data,
isolation
mode.
And
so
you
can.
A
A
A
A
Yeah
and
so
they
claim
low
overhead
and
maybe
the
ability
to
scale
out
and
so
yeah.
This
is
a
no
instrumentation
approach,
where
it's
not
quite
no
instrumentation,
but
it's
very
low
instrumentation.
At
least
you
do
have
to
build
with
symbols.
I
was
seeing
somewhere
or
only
parts
of
it
work,
but
it
will.
A
What
you'd
use
for
prometheus,
so
you'd
normally
with
prometheus,
have
to
kind
of
build
in
an
endpoint
and
and
apparently
they
also
can
work
with
openssl
and
go
tls
to
give
you
the
encrypted
payloads.
A
C
A
Ssl
tls
traffic:
oh
we're
back
on
the
pixie
lab
site.
That's
okay
doesn't
currently
ingest
data
from
prometheus
but
would
like
to
and
yeah
so
install
we're
going
to
use
mini
cube.
I've
already
got
a
mini
cube.
Cluster
set
up.
A
I
feel
like
damon
sets,
made
agents
a
lot
less
invasive
back
in
the
back
in
2000
we
didn't
when
we
didn't
have
containers
and
you
had
to
install
the
agent.
You
had
to
worry
about
the
library,
the
library
dependencies
of
the
agent,
as
well
as
the
library
dependencies
of
your
actual
application
and
when
they
com
and
when
they
collided,
then
you
were
really
sad
yeah.
If
this.
A
If
this
only
sent
data
to
the
cloud,
it
would
be
less
interesting,
although
if
you
could
run
your
own
cloud
for
sending
the
data
to
that
seems,
like
would
also
be
pretty
cool.
So
let's
see
local
kubernetes.
A
A
Use
but
they
support
the
big
three
clouds
and
self-hosted
with
some
caveats,
and
so
we
are
going
to
be
using
a
mini
cube
environment.
It
looks
like
they
don't
support,
kinder
docker
desktop,
which
is
interesting.
B
A
We've
got
an
eight
gig
machine.
Oh
this
is
interesting.
It
only
supports
x86
64.
no
arm
so
and
only
supports
linux.
I'm
guessing
that's
again
for
the
ebpf
and
they've
tested
these
distros.
A
B
Start
because
we
want
to
wait,
it's
dusky.
B
A
A
B
A
Okay,
I
just
didn't
copy
over.
A
So
I'll
get
my
my
linux
cube
can
fix
that
up
in
just
a
moment
and
we'll
go.
A
Docs
one
thing
they
don't
have,
as
far
as
I
can
see,
is
a
link
for
how
to
actually
update
the
website,
which
is
kind
of
nice.
I
noticed
some
other
sites
are
having
where
it's
like.
Okay
click
here
and
you
can
edit
that
exact.
A
A
A
Sorry
I
thought
I
had
my
cube
config
set
up,
but
I
only
have
it
on
windows
and
if
we
look
at
the
install
it's
all
bash
scripts,
so
I'm
gonna
need
to
figure
that
out-
and
I
just
don't
want
to
show
off
all
of
my
sort
of
my
certificate
stuff.
Oh
and
I'm
not
sharing
my
screen
even
better.
A
A
See
yeah
this
is
this
is
windows.
I
have
windows
subsystem
for
linux
and
I'm
using
mini
cube
on
windows.
I
like
I
like
getting
that
experience
of.
What's
it
like,
and
so
this
is
the
first
hurdle.
I'd
probably
feel
if
I
was
a
windows
developer
is
oh
look,
here's
a
bunch
of
bash
stuff
and
I
have
to
interrupt
it
nicely
if
you
haven't
done
it
before
mini
cube.
Has
some
nice
okay,
if
you're
using
you
know
linux,
do
this?
If
you're
using
mac,
do
this
you're
using
windows?
A
B
B
A
I
do
have
vs
code
here.
We
go.
B
B
A
A
Just
a
sec,
I'm
switching
up
the
paths
in
my
cube
config
and
I
think
at
this
point.
A
This
point:
I'm
not
going
to
have
any
secrets
on
here,
so
that's
one
annoying
thing
about
cubeconfig
is
it's
not
something
you
can
go
and
edit
live,
or
at
least
not
on
a
recording,
unless
you
want
to
rotate
all
your
keys.
That
doesn't
really
sound
like
my
idea
of
a
fun
afternoon.
B
I
like
see
what
I'm
gonna
do.
Well
that
says
darwin,
so
I'm
gonna
need
to
switch.
B
A
D
B
A
A
A
B
A
B
They
using
they're
using
logaris
and
they're.
B
A
See
here's
load
test,
it
looks
like
it's
still
in
init,
it
looks
like
most
of
the
other
stuff
is
running,
but
not
all
of
it's
ready.
B
A
See
this
did
not
work
user's,
not
registered.
Please
sign
up.
A
I'm
also
wondering
why
this
token,
at
the
end,
has
a
hyphen
that
looks
a
little
cut
off.
So
if
you
remember
why
you
know
how
this
works
there
is.
Where
are
we
back
here?
A
There's
a
portion,
that's
on
the
cluster
and
there's
a
portion
that
can
report
to
the
cloud
and
if
you
go
back
and
look
at
px.dev,
there's
a
bug
about
self-hosted
install
script
coming
soon,
and
so
currently
there
is
a
hosted
thing
and
the
code
used
for
hosting
is
open
source,
but
it
does
try
to
hook
you
up
to
the
cloud
and
it
says
the
self-hosted
will
have
no
external
dependencies,
but
you'll
have
to
manage
it
yourself,
and
it
says
I
wanna.
A
I
want
self-hosted
pixie
now
and
here's
some
stuff
for
doing.
D
A
B
See
pixie
lab
city,
I
sign
up
sign
up
with
google.
B
A
A
But
it
looks
like
they're
working
on
a
completely
self-hosted
version,
but
yeah
the
fact
that
you
need
a
cloud
connector
is
you
know,
I
don't
think
that's
the
long-term
plan,
but
right
now,
yeah.
B
B
C
B
A
A
You
think
it
was
an
let's
see.
B
A
This
is
with
eight
gigs
of
memory
and
limits.
Nowhere
close,
oh,
but
I'm
noticing
that
all
of
this
pixie
stuff
is
making
zero
requests.
B
Let's
see
see
if
I
can
make
this
bigger.
A
It
failed
to
create
pod
sandbox
error,
while
dialing.
Let's.
B
A
And
that
huge
java
app
deployment
well,
so
this
is
in
an
8
gig
vm.
A
C
A
Yeah
and
as
I
was
saying,
we've
got
eight
okay.
Maybe
I
should
spin
up
a
new
mini
cube.
Give
me
a
moment.
A
Oh,
is
there
a
simple
demo,
I've
got
32
gigs
of
ram,
so
I
could
spin
up
a
new
mini
cube
or
I
could
do
a
smaller
demo.
B
A
B
B
B
A
Go
go
mini,
cube
yeah,
so
while
this
is
kicking
off
and
pulling
the
image
and
all
that
stuff,
I
thought
maybe
I
would
go
back
and
look
at
the
demo
stuff
and
see
how
it
was
doing
that.
C
A
Oh
cluster,
I
could
spawn
a
gke
cluster
too,
but
usually
mini
cube
doesn't
take
too
long
here
we
go.
C
B
B
B
B
B
D
B
B
D
A
We
should
have
plenty
of
space,
then,
let's
see.
A
I
don't
see
as
many
databases
here,
so
we
have
a
lot
of
different
services,
but
almost
no
backing
databases
that
might
be
taking
up
a
bunch
of
space.
So,
let's.
B
C
B
A
A
A
And
access
data
without
being
the
same
vpc
and
network,
and
so
when
we
look
at
this
here.
C
A
A
Okay,
so
here's
the
other
one
and
I
can
see
that
it
is
in
pass-through
mode.
If
I
want
to
be
in
data
isolation
mode,
I
can
switch
this
and
we
go
directly
to
the
module
in
your
cluster.
A
A
A
This
is
now
going
to
try
to
connect
to
my
cluster
and
it's
probably
just
going
to
spin,
because
I
don't
have
pass-through
mode
and
I
would
need
to
get
need
to
get
the
network
set
up
there
and,
let's
see.
A
So
this
is
your
browser,
so
actually,
let's
go
back
and
switch
it
back
to
pass
through
false.
A
A
Stop
so
we're
making
these
graphql
queries.
B
A
Start
time
so
this
is
all
going
to
with
pixie.ai.
A
Somewhere
in
here
I'm
here
we
go:
oh
no!
That's
lunch,
darkly
health
check.
So
now
this
is
trying
to
hit
this
url.
This
192
168
url,
and
I
bet
that
if
we
look
over
here
and
we
do
a
cube,
control,
get
svc
dash,
lpl.
A
No,
these
are
all
10
space
addresses,
but
this
looks
like
this
is
the
address
that
it's
trying
to
pick
up.
I'm
not
sure
where
it's
getting
that.
A
C
A
And
so
I
would
need
to
set
up
forwarding
all
of
that
to
make
it
work,
and
so
it's
nice
that
they
can
proxy
you
through
the
cloud
and
do
sort
of
a
reverse
pattern.
It
looks
like
it's
also
possible
to
set
up
locally,
since
I'm
interested
in
figuring
out
everything
else.
That
was
quick.
I
don't
know
if
you
noticed
that
I
switched
it
back.
A
And
all
of
a
sudden
it
popped
up,
and
I
have
a
pixy2
node
and
I
can
look
at
my
cpu
usage-
seems
really
high.
Oh,
but
I
bet
I
have
a
load
test
running.
Let's
see.
A
A
A
We
can
group
things
by
namespace,
so
we
can
see
that
almost
all
of
our
cpu
load
is
coming
from
px
online
boutique,
but
we
are
spending,
I
guess,
about
20
percent
of
our
cpu
keeping
track
of
all
of
the
work
going
on
by
dread
by
territon
virtual
memory
size.
And
if
you
remember
back
here,
this
stuff
is
actually
getting
proxied
through
the
cloud
to
the
vizier
on
the
cluster.
So
all
this
data
isn't
actually
getting
pushed
up
into
the
cloud.
A
B
A
A
A
Yeah,
the
the
short
form
is,
I
think
all
of
this
is
actually
being
transformed,
so
select
your
cluster.
A
Run
the
script
using
the
run
button
running
the
script
you
should
see,
oh,
which
one
is
this
px
cluster
looks
like
cluster
is
the
one
that
you
wanted
to
see.
B
B
I'm
not
sure,
let's
see.
A
If
we
looks
like
you,
can
cluster
things
by
service
or
not,
let's
see
if
we
look
at
px
online
boutique,
we
get
a
list
of
pods.
I'm
not
sure
why
we're
not
getting.
B
A
B
Let's
see
developer
tools,
yeah,
not
network
anymore,.
A
Well,
these
are
all
warnings
field
to
live
resource,
oh,
but
this
might
be
a
while.
A
A
Pixie
audio
hijack,
I
don't
think
so.
A
Great,
I
have
no
idea
what
happened
there.
Technology
is
a
mystery
for
now.
I'm
wondering
why
this
px
cluster.
A
But
so
yeah
it
looks
like
you
can
write
your
own
scripts
to
do
your
own
representation.
A
B
A
And
over
here
in
variables.
So
if
I
do
something
like
this
name.
B
B
A
Then
I
guess
I
need
to
hit
run
again.
Oh
nope,
when
I
ran
it,
it
removed
that
follows
pan
follows
pandas,
it's
a
subset
of
python.
So
it's
nice,
it's
nice,
that
it's
a
subset
of
python,
so
people
can
reuse
that,
rather
than
sort
of
having
to
learn
their
own
language,
so
that's
kind
of
nice.
B
B
A
So
here's
the
demo
script,
let's
run
it.
Oh,
this
is
kind
of
interesting,
so
we
can
actually
see
individual
http
requests.
You'd
want
to
be
careful
with
giving
this
giving
this
out
because
you
could
have
pii
in
here.
A
A
Yeah
it
looks
like
this
was
an
existing
new
relic
service
and
that
they
are
they're
open
sourcing.
It
so.
A
I
thought
that
was
pretty
neat,
so
this
is
interesting,
so
this
is
actually
giving
us
an
overview
into
all
the
dns
queries
going
on
in
our
cluster.
So
we
can
see
we're
looking
up
this
name
and
we're
getting
back
the
answer,
and
so,
presumably,
if
we
wanted
to
we're
getting
stuff
from
dns
events.
B
B
A
A
Well,
I
think
this
is
a
different
model.
This
is
more
of
a
data
processing
model,
carlos
than
a
like
a
query
language
you
can
think
of
grafana
as
more
of
a
more
of
like
a
sql
standing
query.
A
B
B
What
the
other
stuff
was,
I
thought
you
should
allow
it
to
modify
the
html,
not
sure
I
follow.
A
Set
up
the
script
environment,
let's
see
data
frame.
B
A
If
you
notice
this
dns
stuff
right
now
has
a
bunch
of
stuff
in
it,
including
the
request
body.
But
I
wonder
if
we
can
actually
say.
B
A
Okay,
so
this
would
give
us
just
the
queries.
I
don't
know
enough
pandas
to
be
able
to
filter.
A
Bringing
in
pandas
does
feel
pretty
cool
it
does.
It
is
kind
of
like
a
like
a
jupiter
notebook.
A
B
B
A
I
don't
know
how
to
take
apart
the
json
from
hosting
inside
the
pod.
Oh
I'm
pretty
sure
that
it's
recording
the
dns
queries.
I
was
hoping
to
be
able
to
filter
things
to
only
this
online
boutique
or
only
like
cube
system.
A
Oh,
this
is
cool,
so
it's
got
this
contextual
information.
We
can
also
do
a
group
by
and
build
aggregates
out
of
that.
B
B
A
Work
yeah,
so
this
is,
this
is
a
subset
of
python.
It's
not
actually
python.
It's
python
compatible
pandas
looking
stuff,
and
so
I
need
to
call
this
px.display
to
show
things,
but
I
might
be
able
to
say.
B
B
B
C
B
Was
hoping
for
pod.
C
A
This
looks
like
something
where
I
would
have
to
yeah
yep
pandas
to
go
back
to
the
http
data.
Query,
for
example,
pandas.
A
On
this
service
column
to
actually
do
a
filter
and.
C
D
C
B
C
D
C
A
D
C
A
D
B
A
Okay,
so
this
is
all
live
stuff,
but
I
think
you
can
also.
A
A
And
say
you
know:
hey,
you
know
log
every
time
that
this
happens
and
it
goes
in
and
dynamically
instruments
the
cluster.
For
that.
B
A
But
yeah
this
is
a
pretty
neat
capability
to
actually
add,
add
debug
logs
or.
A
Yeah,
so
they
have
some
other
things
like
that.
Like
database
query,
profiling.
B
B
A
Yeah,
I
don't
know
where
that
collab
notebook
went
so
we'll
have
to
dig
it
back
out,
but
it
looks
like
you
can.
You
can
pick
out
individual
slow
queries
do
request.
A
Okay,
so
for
those
of
you
who
weren't
aware
pixie
is
pretty
new
and
I
think
we're
seeing
that
in
some
of
this
documentation
monitor
the
health
of
your
services,
and
this
also
points
to
a
co-lab
notebook.
That's
no
longer
there.
B
A
But
the
nice
thing
about
having
this
in
a
language
is
that
you
can
also
like
check
these
things
in
and
share
them
around.
So
I
really
like
that
you
can
get
these.
You
can
see
the
arguments
get
reflected
up
here
as
links.
A
One
thing
that
I
would
suggest
adding
is
the
ability
to
set
the
end
time,
which
is
really
handy.
If
you
do
have
long
term
retained
data
to
be
able
to
say
during
this
outage
link,
we
did
this,
and
people
can
go
in
and
look
at
the
actual
data
and
you
know
go
back
and
see
if
there
would
be
a
better
way
to
detect
the
problem,
and
then
I
like,
I
like
that,
it's
using
you
know
pandas
rather
than
like.
Oh,
we
made
up
yet
another
language
for
you
to.
D
A
But
yeah,
I
think
I
think,
I'm
about
the
end
of
what
I
can
explore
right
now.
I'm
glad
that
we
got
things
working.
A
A
A
We're
only
using
like
four
gigs
of
memory-
and
you
know,
I'd-
want
a
little
extra
space,
but
the
other
one
I'm
guessing
is
a
lot
bigger.
A
Let's
see
so
prometheus
is
basically
a
metrics
aggregation
system,
so
you
need
to
instrument
your
code
ahead
of
time
to
collect
all
of
these
metrics
and
then
prometheus
does
a
great
job
of
storing
them
historically
and
letting
you
know
okay,
six
months
ago
or
a
year
ago
you
know
my
data
looked
like
this,
and
now
it
looks
like
this,
whereas
pixie
seems
like
it's
a
little
bit
more
of
a
debugging,
an
app
instrumentation
system
which
isn't
surprising,
given
the
folks
at
new
relic
have
been
doing
that
for
many
years,
and
so
it
looks
like
it's
a
lot
easier
to
dig
into
individual
data
points
and
I'd
actually
be
really
curious.
A
What
the
underlying
data
structures
they're
using
are,
it
looks
like
it's
more
likely
to
be
sampled
traces
than
aggregated
metrics,
so
prometheus's
model
of
the
world
is
basically
things
happen
and
you
increment
counters,
and
your
counter,
you
know,
goes
up
over
time
or
maybe
it's
how
much
disk
space
you're
using,
and
so
you
get
a
number
for
that.
And
then
it
does
a
bunch
of
collection
on
numbers
and
let
you
group
them
up
by
labels
and
do
math
with
that.
A
A
A
You
create
a
connection
and
you
call
read
or
write
on
the
connection.
They
can
actually
hook
that.
Oh
nobody
can
see
what
I'm
seeing
they
can
actually
hook.
A
These
read
and
write
calls
in
the
library,
and
so
they
can
actually
collect
what
you're
sending
over
the
tcp
links
because
they've,
if
you
hook
the
go
tls
and
you
replace
all
of
the
methods
on
connection
with
thunks
that
record
what
you're
doing
and
then
you
can
pick
up
the
net
connection
and
you
can
pick
up
the
reads
and
writes
and
you
can
actually
trace
over
tls,
which
is
really
clever
and.
A
Yeah
it
could,
it
could
probably
intercept
the
mutual
tls
for
istio
if
you're
using
istio
istio
can
also
actually
has
instrumentation.
That
can
do
the
same
thing.
They've
added
that
instrumentation
into
envoy
manually,
but
the
clever
thing
that
pixi
is
doing
is
that
they're
actually
they're
actually
automatically
doing
it.
A
They're,
also
using
ebpf
they're,
basically
creating
a
whole
bunch
of
different
trace
points
in
different
parts
of
the
program,
and
so
there's
some
trace
points
that
are
going
through
the
kernel
and
there
may
be
some
trace
points
that
are
in
the
application
itself,
and
you
can,
you
know,
put
a
whole
bunch
of
different
trace
points
in
there
record
those
traces
and
then
see.
Okay.
A
You
know
these
things
happened
and
then
they
happen
in
the
same
thread.
So
I'm
pretty
sure
that
you
know
that
it's
all
part
of
the
same
thing.
The
really
interesting
thing
is
for
golang
ebpf
on
its
own
may
not
be
sufficient,
because
golang
uses
green
threads.
A
Oh
ebpf
can
do
trace
points
in
user
space.
I
didn't
know
that
cool
okay,
so
yeah
they're,
probably
using
ebpf,
for
that
they
are
probably
also
having
to
hook
the
go
thread.
Implementation
because
all
your
go
all
of
your
go
processes
or
your
your
go
functions.
Your
go
routines,
that's
the
word,
I'm
looking
for
all
your
go.
A
Routines
are
multiplexed
onto
gomax
procs,
so
onto
like
four
or
16
threads,
or
something
like
that,
and
so
they
actually
have
to
keep
track
of
the
context
switches
that
go
is
doing
in
user
space
and
it
looks
like
there
are
some
directions
on
how
to
get
started.
A
They
have
some
development
instructions
on
how
to
get
everything
set
up,
haven't
tried
doing
this
yet
but
yeah.
This
is
a
really.
A
This
is
a
really
clever
set
of
stuff
and
it
looks
like
they
are
looking
to
collect
a
lot
of
different
data.
A
For
those
of
you
who
aren't
familiar
with
it,
google
has
a
stackdriver
debug
technology
that
is
a
bit
similar,
but
it's
all
in
their
closed
source
system
and
it
works
not
quite
the
same
because
it
actually
integrates
as
a
library.
A
So
it's
pulling
things
out
from
your
from
your
application
binary,
rather
than
pulling
it
out
from
the
kernel
side,
but
yeah
you
should
we
should
be
able
to
get
trace
details
of.
A
A
B
B
B
A
Waiting
for
the
url-
oh
that's
an
init
container,
never
mind!
That's
not
the
interesting
one.
A
B
A
B
B
A
This
may
be
hard-coded,
I'm
gonna.
Let
someone
else
dig
in
and
figure
out
how
to
actually
get
it
to
trace
kubernetes
stuff,
but
it
looks
like
it
probably
should
be
able
to.
C
A
I
don't
know
what
this
thing
is.
That's
curl
and
this
is
pixie,
proud,
vizier,
kelvin
image.
A
It's
a
thing
it
does
stuff,
I
don't
think
I'm
gonna
get
to
the
bottom
of
it
in
the
next
five
minutes.
So
I
think
this
is
probably
going
to
be
where
I
cap
it
off,
but
yeah
something
to
keep
an
eye
on.
I'm
particularly
looking
forward
to,
and
I'm
probably
going
to
watch
this
issue
here
for
self-hosted
pixie
install
script,
because
I'd
really
like
to
run
this
on
my
own
environment
without
tethering
it
to
a
cloud.
A
I
like
that,
for
just
learning
how
stuff
works
so
hope
you
all
are
having
a
lovely
lovely
day
and
a
good
evening
or
afternoon
or
whatever
you've
got
it's
sunny
out
here
still,
so
I'm
gonna
get
a
little
more
of
that
sunshine.