►
From YouTube: Reproducing Production Issues in your CI Pipeline Using eBPF - Matthew LeRay & Omid Azizi
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Reproducing Production Issues in your CI Pipeline Using eBPF - Matthew LeRay, Speedscale & Omid Azizi, New Relic
Observing production workloads with enough detail to find real problems is difficult, but it's getting easier with the community adoption of eBPF. As the technology becomes better understood, tools like Falco, Cilium and Pixie are increasingly appearing in production clusters. But have you ever considered using eBPF data to help with unit tests, Continuous Integration and load testing? This talk will explain the basic technology behind eBPF while presenting some examples of how to use data collected via eBPF for a variety of software quality use cases. We'll use the Pixie CNCF sandbox project to pull data and replicate production issues on the developer desktop for debugging. You'll also get some ideas on using those calls in your Continuous Integration pipeline to sanity check builds before they are deployed. Included in that discussion will be handling some common issues like timestamp skew and authentication. All examples are open source and available after the talk.
A
All
right
welcome
everyone.
We're
really
happy
to
be
here
today
to
talk
about
reproducing
production
issues
in
your
ci
pipeline
using
ebpf,
so
yeah.
My
name
is
omid.
I
work
at
new
relic,
I'm
a
engineer
there
principal
engineer.
I
was
a
founding
engineer
at
pixie,
so
I've
spent
quite
a
bit
of
time
working
on
observability,
getting
data
with
ebpf
and
we're
going
to
talk
a
lot
more
about
that
in
the
talk-
and
I'm
super
excited
to
be
here-
matt,
hey
so.
B
A
B
We
did
better
all
right,
okay,
so,
like
I
said
I,
I
hate
writing
tests
and
probably
you
do
too
one
of
the
things
we
did
in
previous
generations
of
technology
is.
We
would
go
and
write
these
handcrafted
tests
where
we
would
go
and
make
sure
it's
going
to
work.
We
would
try
to
guess
problem
is.
Is
we
still
have
huge
numbers
of
production
incidents?
So
what
I
kind
of
was
trying
to
accomplish
with
this
talk
is
give
everybody
a
bunch
of
easy
things.
B
You
can
do
right,
github
repositories,
you
can
go,
look
at
that,
will
show
you
how
to
in
how
to
instrument
your
application
and
run
realistic
test
scenarios
using
traffic
replay
in
your
continuous
integration
pipeline.
So
that
is
that's
that's.
What
we're
trying
to
achieve
is
get
more
tests,
better
tests
with
automation,
not
with
human
intervention.
B
So
what
is
traffic
replay
now?
You
may
have
heard
of
this
before
it's
been
around
the
industry
a
little
bit,
but
basically,
when
it
comes
to
software
quality
and
validation,
it
means
treating
your
tests
like
cattle,
not
pets.
So,
instead
of
having
those
hand
curated
hand
curated
tests,
much
like
your
kubernetes
infrastructure,
you
also
you
want
to
have
them
continuously
refreshed.
You
want
to
just
blow
them
away,
get
rid
of
all
the
tests
when
they're
not
not
valid
anymore,
and
you
want
to
treat
it
as
a
system.
B
That's
continuously
updating,
instead
of
as
this,
this
sort
of
written-
and
you
know
cast
in
concrete
sort
of
thing,
so
a
couple
different
approaches
to
software
quality
that
every
well
many
people
probably
are
using
if
you're
an
early
adopter
of
kubernetes
and
cloud
native,
you
probably
have
tried
to
test
and
prod,
which
I
like
to
live
dangerously
too.
That's
great.
I
do
a
lot
of
testing
in
prod,
sometimes
by
accident,
and
but
there
are
some
limitations
to
doing
that.
B
So
you
know
live
dangerously,
so
companies
that
were
very
early
adopters
like
twitter
came
up
with
an
idea
of
traffic
mirroring.
Basically,
the
idea
of
traffic
mirroring
is
to
apply
the
scientific
method
to
testing
new
releases.
So
the
process
goes.
You
have
a
continuous
integration
pipeline,
you
go
and
you
deploy
it,
but
you
try
to
limit
the
blast
radius
of
a
failure.
So
you
got
this
new
code,
you're,
not
sure
if
it's
going
to
work,
so
you
deploy
it
along
with
two
replicas
and
then
you
go
and
see
how
it
performs
versus
the
replicas.
B
So
you
get
kind
of
this
comparison
chart
you
know
to
see
what
is
you
know?
What's
working
and
what's
not
working
now
that
works
okay
for
twitter,
because
twitter
has
a
certain
set
of
conditions
that
you
may
not
have
the
first
one
is
they
have
billions
of
transactions
a
day,
I'd
actually
maybe
trillions,
but
billions
at
least
a
day.
You
know
tons
and
tons
of
traffic
constantly
going
through.
So
if
they
have
a
small
amount
of
error,
that's
okay
right,
because
somebody'll
just
hit
refresh
no
big
deal.
B
Now,
if
you
work
at
deutsche
bank
totally
different
story,
you
can't
have
these
failures
right.
So
that's
the
limitation
of
traffic
mirroring
and
there's
one
additional
limitation
is
time.
Traffic,
mirroring
only
shows
you
what's
going
on
in
your
production
system
at
that
time,
the
time
in
which
you're
mirroring
so
to
solve
these
problems.
B
The
new
new
generation
of
the
technology
is
to
actually
use
traffic
replay
with
persistent
storage.
So
the
idea
is
we're
gonna,
we're
gonna
tap
into
production,
usually
using
a
proxy
which
we'll
get
to
that
in
a
second,
usually
using
a
proxy
we're
gonna
tap
into
production,
we're
basically
going
to
open
a
big
pipe
and
for
some
period
of
time
we're
going
to
funnel
that
traffic
into
a
data
store.
Then
we're
going
to
try
to
reuse
it
either
on
our
desktop
or
using
our
ci
system.
B
So
the
idea
is,
you
can
constantly
take
these
new
snapshots
right
and
constantly
re-run
them
using
persistent
storage,
so
even
in
the
middle
of
the
night,
still
get
good
traffic
right
in
your
test
environment
or
let's
say
you
have
a
one-time
event
that
explodes
right
in
production.
You
can
go
capture
that
traffic
and
then
use
it
again
and
again
in
ci
to
make
sure
that
never
happens
again
so
stuff
like
that.
So
that's
the
last
one
which
is
traffic
replay,
which
is
what
we're
going
to
focus
on
in
this
talk.
A
Cool,
so
if
you
want
to
do
traffic
replay,
where
do
you
start?
Obviously
you
need
to
record
information.
You
need
to
capture
some
of
the
traffic
right.
You
need
to
know
what
every
single
service
in
your
kubernetes
every
single
pod
in
your
kubernetes
cluster
is
saying
to
every
other
pod
right.
If
you
want
to
know
what
traffic
is
hitting
a
service,
you
need
to
be
able
to
trace
that
information
right.
A
The
challenge
is:
how
do
we
do
that?
How
do
we
do
that
on
a
production
cluster
in
an
untrue,
unobtrusive
way
in
a
safe
way,
in
a
way
that
doesn't
break
the
system
so
that
we
can
get
realistic
traffic
patterns
that
we
can
then
replay
in
our
ci
pipelines
right
and
that's
where
the
ebpf
solution
is
going
to
come
into
play
right?
A
So
I
want
to
introduce
you
to
pixi
pxe
is
an
ebpf
based
tracing
solution
on
kubernetes
we're
going
to
talk
a
little
bit
about
ebpf
and
what
it
is,
but
pixi
itself
is
an
open
source,
cncf
observability
platform
and
among
the
various
different
observability
things
that
it
gives
you.
One
of
them
is
that
it
automatically
traces.
A
A
A
So
if
you've
heard
about
ebpf
before
and
if
you
haven't,
you
should
go,
read
more
about
it,
but
if,
if
if
you've
heard
about
it
before,
sometimes
you
may
have
heard
of
it
as
a
you
know,
a
vm
in
the
kernel,
sandbox
environment,
all
that
sort
of
stuff
I'm
going
to
put
those
are
great
descriptions-
are
accurate.
I'm
going
to
put
those
aside
for
a
moment,
and
I
just
want
to
tell
you
what
you
can
do
with
it
right,
and
so
I
like
to
use
an
analogy.
A
A
You
can
go
ahead
and
put
a
breakpoint
anywhere
in
your
code
and
say
when
you
reach
this
line
of
code
break
right
and
then
you
get
a
you
get
a
kind
of
a
terminal
and
then
you
can
go
poke
around.
You
can
go,
read
the
values
of
different
variables.
You
can
read
the
memory.
You
can
inspect
everything
that
you
want
right,
so
pretty
powerful.
A
Of
course,
if
you
try
to
put
a
debugger
into
production
and
break
it,
you're
going
to
halt
the
service,
and
that's
no
good
right,
so
we
can't
do
that.
But
what
ebpf
allows
you
to
do
is
essentially
set
a
similar
breakpoint,
which
is
called
a
probe,
and
then
you
kind
of
automate
the
reading
of
the
different
variables
that
you
want
to
get.
A
So
you
can
go
in
and
write
an
ebpf
script
that
says:
stop
when
you
reach
this
line
of
code,
go
collect
the
value
of
the
request,
variable
or
the
fields
of
that
variable
or
whatever
else,
other
information
that
you
want.
Take
that
out,
and
you
know,
push
it
out
into
user
space
or
put
it
somewhere
so
that
you
can
kind
of
go,
get
it
later
and
then
immediately
resume
the
execution
of
the
program.
A
It's
very
quick,
it's
very
efficient!
It's
safe!
The
kernel
will
actually
check
that
your
ebpf
program
won't
kind
of
mess
up
the
kernel
or
anything
like
that,
and
so
you
can
really
in
an
unobtrusive
way.
That's
the
key
word
grab
information
about
what's
happening
in
your
system,
so
that
you
can
collect
it
for
replay.
A
Yeah,
that's
a
great
question,
so
I
mean
ebpf
itself
is,
is
very
lightweight
like
you,
unless
you're
putting
it
in,
like
the
tightest
loop,
you're,
probably
not
going
to
notice
any
impact
to
your
performance,
the
probe
itself
is
going
to
be
so
quick.
You
won't
notice
it
it's
negligible
in
terms
of
pixie
itself.
A
Pixie
takes
because
it
does
other
processing
and
parsing
of
information
takes
anywhere
between
two
to
five
percent
of
your
your
application,
but
the
ebpf
part
itself
is
really
lightweight
yeah
and
then
I
have
an
example
here,
I'll
just
walk
through
that
quickly
so
like
here
we
have
a
process
request
function,
that's
your
application
code.
Let's
say
you
can
put
a
probe
on
the
send
pong
function.
You
say
every
time
I
reach
the
senpang
function.
A
I
want
to
do
something
and
then
the
ebpf,
that's
your
trigger,
and
the
ebpf
program
here
is
the
most
simplest
one
you
can
think
about
is
just
counting.
It's
saying
how
many
times
did
I
reach
this,
but
you
could
also
go
in
and
say
what
was
the
value
of
the
variable?
What
was
it?
Was
it
an
http
request?
What
was
the
content
of
the
http
request?
All
the
sorts
of
information
that
we
would
want.
A
Okay,
so
that
seems
pretty
powerful
ebpf
again.
I
really
encourage
folks
to
go,
read
up
more
about
it.
If
you,
if
you're
not
familiar
with
it,
it's
super
cool
pixie.
The
way
that
pixi
uses
evpf
and
his
protocol
tracer
is
that
we
want
to
be
able
to
trace
all
the
traffic
in
your
system
right
and
the
important
thing
is.
We
want
to
be
able
to
trace
the
traffic
in
your
system,
regardless
of
what
language
you
wrote
it
in,
so
python
go
c.
A
Whatever
ruby,
we
don't
care,
node.js
right,
no
matter
what
sort
of
application
you're
writing
we
want
to
be
able
to
trace
that
traffic
and
not
have
you
change
anything
about
your
code
right.
So
what's
the
best
place
for
us
to
put
the
evpf
probe
on
is
actually
in
the
kernel
itself,
so
you
can
do
that
with
ebpf
as
well.
So
you
can
go
into
the
linux
kernel
and
say
when
the
application
makes
a
send
syscall
right
trigger
trigger
an
ebpf
probe
right,
and
so
then
it
doesn't
matter
what
framework
you've
written
your
application
in.
A
We
can
trace
it
right
and
so
every
time
a
message
is
sent
we
can
trigger
on
that
and
we
can
collect
the
information
about
what
was
sent
every
time
that
a
message
is
received.
We
can
trigger
on
that
and
we
can
collect
information
about
the
response.
So
an
http
message
goes
out,
we'll
get
it
if
it's
a
mysql,
we'll
get
it.
A
It
doesn't
matter
as
long
as
it's
going
through
the
linux
kernel,
which
everything
has
to
we'll
get
it
is
it
in
a
container
sure,
doesn't
matter
we'll
get
it
because
the
container
sits
on
top
of
the
linux
kernel
and
that
traffic
eventually
goes
through
the
linux
kernel
all
right.
Once
we've
gotten
the
data
right,
the
edpf
probe
collects
that
stuff
puts
it
in
memory
somewhere
and
sends
it
up
to
the
pixie
edge
module
essentially
sends
it
up
to
the
rest
of
pixi,
where
pixi
will
parse
the
data
put
it
in
a
structured
format.
A
Now,
some
of
you
might
be
thinking,
oh
yeah,
but
my
application,
my
application's
tlsed
right.
So
by
the
time
it
reaches
the
kernel,
it's
all
encrypted,
so
you're
going
to
be
out
of
luck.
How
are
you
going
to
get
the
data
out
there
ebpf
to
the
rescue
again
right,
so
you
can
do
a
lot
of
cool
things
with
ebpf.
A
So
when
you,
when
pixie
detects
that
there's
encrypted
traffic,
what
it
actually
does
is
it
puts
probes
instead
of
on
the
linux
kernel
itself,
it
puts
it
on
the
tls
library.
So,
for
example,
if
you
instrument
open,
ssl
and
probe
every
time,
there's
an
ssl
read
or
an
ssl
write,
it's
essentially
the
equivalent
of
send
and
receive
right
all
the
traffic.
That's
going
through
your
open
ssl,
regardless
of
which
application
it's
coming
from
openssl
is
typically
a
shared
library.
A
A
There
will
be
one
instance
of
that
on
every
single
node
of
your
kubernetes
cluster
right,
and
so,
if
it's
sitting
on
every
node
of
your
kubernetes
cluster
and
it's
watching
the
kernel,
everything
that
is
happening
in
your
cluster
is
getting
monitored
right.
So
we'll
capture
all
the
traffic
from
ebpf,
as
I
mentioned
before,
every
pod.
So
it's
one
instance
of
the
pem
is
monitoring
all
the
pods
that
are
sitting
on
all
their
containers:
everything
it's
all
getting
traced
by
the
pixie
edge
module
characteristics.
A
We
already
mentioned
the
zero
instrumentation,
no
code,
modifications,
that's
the
beauty
of
ebpf
right.
You
didn't
have
to
change
anything.
Other
characteristics
of
pixi
is
that
it
is
a
distributed
architecture.
So
all
the
data
is
held
on
the
edge
in
the
cluster
itself
and
there's
a
way
to
query
the
data
to
get
out
just
the
information
that
you
want.
So
we're
not
sending
volumes
and
volumes
of
data
up
into
the
cloud
or
anything.
A
A
We
didn't
want
to
reinvent
anything:
it's
pretty
much
pandas,
so
you
can
query
your
data
and
get
get
the
information
that
you
want
out.
So
when
you
make
a
query
you're
setting
up,
let's
say
in
the
browser,
you're
you're
working
with
pixi,
you
make
a
query
to
get
certain
data
or
you
you
click
on
something
where
you
want
to
see.
Like
the
service
map,
a
query
will
come
down
through
down
to
the
the
pixi
edge
modules.
The
pxe
edge
modules
will
filter
out
the
relevant
information.
A
A
A
The
first
thing
that
you're
going
to
see
in
the
ui
at
the
top
is
a
service
map,
so
that's
showing
pixi's
recording
all
the
traffic,
all
the
http
traffic
between
all
the
different
pods
and
over
the
last
five
minutes,
in
this
case
that's
configurable,
but
over
the
last
five
minutes
is
seeing
everyone
who's
spoken
every
pod,
that's
spoken
to
another
pod
and
it's
showing
that
information
on
the
service
map.
In
addition
to
that,
we
can
see
the
list
of
services,
we
can
see
the
list
of
pods.
A
You
can
see
things
like
error
rates,
latencies
all
sorts
of
other
good
stuff,
but
what
we're
interested
in
most
is
kind
of
the
traffic
there
that's
driving
this
service
map,
because
that's
what
we
want
to
get.
We
want
to
get
the
raw
data
out
right.
We
want
to
get
all
the
data,
so
we
can
replay
it
in
our
test
environment
drilling
down
a
little
bit.
You
can
see
like
here's.
A
sock
shop
application
is
the
front
end
is
talking
to
the
catalog
service.
A
It's
also
talking
to
this
orders
service.
There's
some
traffic
going
from
orders
to
shipping.
All
this
is
auto-discovered
right.
You
didn't
again
didn't
have
to
change
anything.
You
can
kind
of
see
the
rate
of
traffic
and
the
latency
profile
error
rates.
That
particular
edge
is
in
red
because
it
has
some
errors,
so
that
might
be
worth
looking
into,
but
that's
a
separate
topic,
so
you
can
get
all
this
stuff
out
of
the
box
and
again,
the
key
here
is
that
it's
all
being
driven
by
this
ebpf
magic
pixie
has
this
concept
of
scripts.
A
So
if
you
go
look
there,
you
can
select
different
views
essentially
to
get
different
data,
and
what
we
see
here
is
there's
like
dns
data,
that
you
can
look
at.
There's
kafka
there's
redis
mysql
postgres
whatever,
but
we're
going
to
look
at
http
data
in
this
demo,
so
we
select
http
data
and
now
what
we
see
is
a
table
of
the
most
recent
traffic
in
the
cluster
across
all
the
namespaces
in
your
kubernetes
cluster,
and
so
you
can
see
there's
a
source
column
that
shows
you
where
the
source
pods
are
coming
from.
A
There's
a
front
end
there
that's
communicating
to
a
user
pod.
You
can
see
the
latency.
You
can
see
it's
a
get
to
a
customer.
So
it's
getting
customer
information,
there's
a
response.
Coming
back.
It
was
okay.
You
can
actually
see
the
json
payload
of
the
the
response
as
well.
If
it's
a
post
instruction,
oh
well,
we'll
see
here
sorry
drilling
in
a
little
bit.
You
can
kind
of
see
the
json
body
here
of
there's
an
address.
There's
cards,
there's
different
information
in
here
that
we're
all
tracing.
A
A
I
just
want
to
see
the
requests
that
have
a
response
status
code
of
400
or
higher,
I'm
looking
for
some
errors
right
in
the
cluster
I'll
just
rerun
this
and
what
you
can
see
now
is
the
response
status
like
we're,
seeing
all
the
responses,
all
the
traffic
that
actually
has
errors
in
them
right,
so
that
could
be
very
useful
in
certain
cases
as
well.
Maybe
those
are
certain
things
we
want
to
pull
into
a
ci
pipeline
kind
of
test.
It
out
right
all
right,
so
now
this
time
actually
done
with
the
demo.
A
A
Pixi
has
a
number
of
ways
to
integrate.
There's
an
api
jrpc
api
that
you
can
interface
with.
Generally
the
input
is
a
pixel
script.
You
essentially
provide
the
query
that
you
want
that
chooses
the
sort
of
traffic
that
you
want
to
extract
out
of
pixi.
Again,
you
could
say
all
the
traffic
that
has
errors
or
just
all
the
traffic,
all
the
http
traffic,
whatever
you're
looking
for
and
then
out
on
the
other
side,
you're
going
to
get
the
data.
A
So
there's
two
ways
to
kind
of:
do
that:
there's
a
pixie
api
using
jrpc
interface
that
you
can
do
to
collect
the
data.
Most
people
might
just
want
to
use
the
ui,
but
when
we
want
to
kind
of
automate
things,
that's
going
to
be
very
useful.
The
jrpc
interface
there's
also
a
plug-in
system
that
lets
you
export
stuff
out
if
you
want
to
have
a
more
tighter
integration
with
other
tools.
B
Okay,
thanks
omen.
Okay,
so
ebpf
gives
us
this
like
superpower
right
and
so
I'm
gonna
go
through
a
couple.
Different
use
cases,
one
of
them
to
use
the
superpower
for
good
and
the
other
to
be
a
little
mean
right
to
our
test
application.
So
to
start
with,
put
yourself
in
the
mindset
that
I
think
most
of
us
have
been
in,
which
is
you,
you
know
you're
at
a
company
and
you
are
taking
over
an
existing
piece
of
software
of
service.
B
Let's
say,
and
there
are
certain
requirements
that
have
been
met,
which
are
that
there
are
no
tests.
No
one
knows
how
the
thing
works.
All
the
people
who
wrote
it
they're
gone
right.
You
have
you
have
some
alerting
in
production,
but
it
generates
a
lot
of
noise.
But
of
course
none
of
it
means
anything
right.
It's
all
the
wrong
learning,
and
so,
let's
see
if
we
can
make
a
dent
in
that
problem,
okay,
so
the
way
we're
gonna
do.
B
That
is
by
inserting
we're
going
to
insert
in
into
the
pipeline
some
of
the
traffic
that
is
extracted
through
the
pixie
api
and
once
again,
codes
on
github.
You
can
go
and
modify
it
use
it
for
your
own
purposes.
I
made
a
slightly
simpler
test
application
just
for
ease
of
use,
which
is
the
it's
just
a
gateway
and
a
payment
service
and
a
user
service,
so
nice
and
easy
to
understand.
B
B
So
because
I,
like
all
of
you
very
nice,
I
created
something
called
pixie
to
curl
in
golang.
If
you
know
golang,
that's
like
the
language
of
kubernetes
that
you
can
go
and
modify
for
your
own
purposes,
but
it
pretty
much
works
out
of
the
box
to
go
and
pull
the
data
out
and
generate
these
big
big
curl
scripts.
Okay-
and
you
can
see
some
of
the
examples
here
from
a
real
real
example.
B
So
the
thing
about
this
there's
a
couple
pieces
I'll
touch
on
again
the
wi-fi
you
know,
I'm
doing
this
with
screenshots,
so
sorry
that
I'm
gonna
hit
you
with
a
wall
of
text
for
a
couple
slides,
but
you
need
to
filter
the
data
down.
One
thing
to
keep
in
mind
about
pixi
is
that
it
is
a
distributed
query
system,
their
ap,
the
api,
but
it
doesn't
have
unlimited
capacity.
B
B
I'll
save
you
from
reading
this.
Basically,
all
we're
doing
is
parsing
parsing
the
response
back
that
we're
getting
from
pixi,
then
feeding
it
into
a
standard,
curl
generation
library.
So
each
one
of
those
requests.
I
was
showing
the
pixel
ui.
You
just
get
a
call
statement
for
each
one,
and
then
we
have
the
output
we're
looking
for
okay.
B
So
I
want
to
stop
all
these
new
developers
from
breaking
the
production
of
the
app
when
it
goes
to
production,
so
I'm
going
to
go
ahead
and
put
it
into
a
github
action.
This
is
where
you
turn
it
into
a
system
instead
of
a
one-time
thing
way.
I
do
that
at
least
for
a
lot
of
our
customers
is
we
say,
give
us
a
kubernetes
or
give
us
a
cube
cuddle,
I'm
sorry,
a
kubernetes
cluster
that
you
can
use
for
testing.
B
That's
it
right,
then
go
and
we'll
insert
a
step
in
the
good
get
the
github
workflow
that
will
go
and
deploy
that
deploy,
that
version
of
the
service
and
run
our
curl
command
against
commands
against
it
and
then
fail
if,
if
none
of
them,
if
one
of
them
breaks-
or
it
takes
too
long-
so
that's
so
now-
we've
gone.
We've
got
that
step
in
github
and
let's
see
what
happens
with
our
two
use
cases.
B
Okay,
so
the
first
use
case
I'm
going
to
go
through
is:
can
we
safely
change
the
api
again?
No,
no
one
knows
how
this
thing
works.
It's
too
hard
to
trace
the
code.
You
know
the
code's
been
around
15
years.
Nobody
knows
how
it
works.
So,
let's
go
and
let's
just
try
stuff
right
this
is
they
teach
you
that
in
computer
science
you
know
master's
degrees,
so,
okay,
so
real
simple
again,
if
I
were
giving
a
live
demo,
we're
just
going
to
go,
make
a
little
minor.
Ati
change,
no
big
deal.
B
Let's
see
what
happens
when
we
commit
the
code
now
our
github
action
runs.
That
runs
the
file
the
400
curl
commands,
and
it
finds
out
that
something
failed.
Okay,
so
now
that
developer
can
no
longer
break
your
production
app.
You
can't
you
know
they're
stopped
from
pushing
when
we
go
and
do
a
little
bit
deeper
inspection.
I
just
went
looked
at
the
logs,
you
probably
have
a
centralized
logging
system,
but
I
just
did
it
with
with
cube
cuddle
logs.
B
So
what
we're
showing
here
what's
happening
is
we're
finding
out
that
when
you
changed
that
when
they
change
that
api
right
when
they
change
that
api,
the
old
production
traffic
won't
work
and
it's
gonna
break
when
they
send
it
out
unless
they
change
the
change
the
downstream
service.
So
let's
use
case
one
now,
you
can
make
a
hundred
of
these
different
examples.
Five,
hundreds
whatever,
but
I
made
it
simple
for
for
a
talk
like
this.
B
Okay,
now
we're
gonna
do
something
that
your
developers,
if
you're
the
one
managing
pipeline,
are
not
gonna
like
because
we're
gonna
test
for
performance
and
we're
gonna.
Do
it
in
a
really
cheap
way
right.
Obviously,
in
the
real
world
you
might
have
load
test
scripts
right,
everybody's,
I'm
sure
you
all
have
perfect
load
test
right.
Everybody
right
integration
test,
I'm
sure
it's
great
the
you!
You
know
there
are
some
great
tools
out
there.
There's
k-6,
there's
hay
from
yana
dogan.
B
My
company
has
one,
although
we
give
it
away
mostly,
you
know
whatever
you
got
something
right,
but
those
scripts
are
probably
out
of
date
right,
they
probably
don't
get
touched
very
often,
so
we're
going
to
go
repurpose
what
we
made
turn
it
into
a
load
test
and
it
turns
out.
That's
super
easy
because
of
the
power
of
kubernetes.
B
B
Now
I'm
going
to
use
the
pixi
interface,
because
pixie
is
a
community
project,
it's
free,
you
can
go
install
it
right
and
what
we
find
out
is
that
when
we
start
ramping
up
the
load
against
this
in
our
ci
pipeline,
the
the
the
latency
of
the
service
jumps
over
a
second
now,
depending
on
what
you
work
on.
You
may
think.
A
second
is
great
for
single
service.
Most
people-
don't
they
know
they
need
to
go.
B
Do
some
optimization
the
key
thing
about
what
we
just
did,
though,
is
all
of
this
happened
automatically
every
time
somebody
makes
a
an
mr
or
a
pr.
You
know
we
go
and
we
break
it
if
they
slow
it
down
too
much
right.
So
you
can
kind
of
tune
your
list,
and
now
you
don't
have
to
keep
up
with
it.
You
don't
have
to
write
your
k6
scripts.
You
have
to
do
anything
you're
just
using
what
you're
recording
from
production
and
you
can
refresh
it
periodically.
B
So
those
are
the
two
simple
use
cases
we
have
time
for
now.
I
want
to
address
something
that
I'm
sure
lots
of
you
folks
are
thinking.
You're,
probably
thinking.
That's
fine
for
the
trivial
little
test
app,
but
I've
got
all
this
other
stuff
going
on
right
in
my
systems.
Of
course
you
do
right.
So
you
know
I
don't
have
a
quick
answer
for
that
other
than
to
say
that
there
are.
You
know
you
can
whittle
away
at
this
problem.
B
There
are
some
there's
some
suggested
reading
that
I
we
have
at
the
end
of
the
presentation
where
you
can
see
what
facebook
and
twitter
and
others
have
done
to
handle
things
like
simulating
third-party
transactions
or
running
against
transactions
that
change
things.
You
know
simple
things
you
can
do
is
you
can
filter
for
only
gets
instead
of
posts
right
so
that
you're
not
changing
the
data?
A
Cool
thanks
man,
so
we're
gonna
start
wrapping
up
yeah,
I'm
gonna
start
wrapping
up
here,
so
just
wanted
to
say
as
we're
wrapping
up.
If
you
want,
if
you're
more
more
about
pixie
learn
more
about
pixi
pixi
is
again
a
cncf
sandbox
project,
completely
open
source,
so
you
can
go
play
around
with
it.
The
website
is
there
github.
A
Is
there
as
well?
We
welcome
contributions.
So
if
anyone
wants
to
contribute,
we
would
love
to
have
more
contributions.
Also
give
us
a
visit
at
the
project
pavilion
where
all
the
open
source
projects
are
come.
Come
chat
with
us.
We'd
love
to
love,
to
hear
from
you
about
use
cases
like
this
or
or
others.
B
Yeah
yeah,
so
that's
it
we're
right
on
time
and
here's
some
other
reading
we'll
be
up
here
for
a
little
while,
if,
if
you
want
to
weigh
in
on
the
the
proper
pronunciation
of
cube
cuddle,
I
have
some
of
these
shirts
as
well.
You
can
grab
because
I
have
a
bunch
carried,
but
that's
it
for
us.
So
thank
you,
yeah.
Thank
you
all.