►
From YouTube: When Connections are Magic: Understanding Performance in Serverless - James Burns, LightStep
Description
Join us for Kubernetes Forums Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
When Connections are Magic: Understanding Performance in Serverless - James Burns, LightStep
Observability! Cloud Functions! APIs! What could go wrong?! While researching the performance of object storage APIs there appeared to be custom run time magic happening leading to significant performance differences. Further research showed that it was *not magic* but lead to even more questions.
Working with modern systems means network connections, many of them. Understanding how those connections impact your customer's experience can be difficult. Distributed tracing helps isolate what parts of the system are failing, but when only implemented at the RPC level the reasons for and scope of network induced issues can be lost. See how network level insights can be integrated into distributed traces and hear how to effective practice iterative observability from the specific case of this research to a general framework for investigation.
A
I
am
James
burns,
and
this
is
when
connections
are
magic,
but
it
turns
out.
This
is
actually
when
connections
are
not
magic
when
I
originally
wanted
to
give
this
talk
when
I
originally
started
doing
work
it
at
at
light
step,
I
was
I,
was
doing
research
and
and
as
part
of
that
research
I
wanted
to
do
what
was
essentially
public
eslary,
so
I
previously
operated
systems
at
William,
stitch
fix
and
as
part
of
operating
those
systems
we
had
dependencies
on
on
different
sorts
of
cloud.
A
Ap
is
particularly
storage
in
pipelines,
and
I
saw
some
really
interesting
stuff.
There
I
saw,
for
instance,
a
case
where
the
aggregate
error
rate
for
service
was
0.00001%
and
that
took
down
the
application.
I
was
responsible
for
in
particular
AZ.
So
these
these
sorts
of
situations
where
the
cloud
provider
doesn't
care
about
the
air
cricket
error
rate,
but
you
care
about
a
lot
I'd
seen
those
I'd
experienced
those,
and
so
but
I
couldn't
talk
about
them.
A
A
lot
any
sort
of
detail,
and
so
I
thought
this
would
be
an
opportunity
to
to
present
this
information
to
write
some
open
source
and
demonstrate
some
of
these.
These
practices
not
just
for
instrumenting,
but
also
seeing
the
issues
and
as
part
of
that
I
want
to
you,
know,
thoughts
and
buzzwords
like
observability
cloud
functions.
Api
is
I
ain't,
and
so
what
I
did
was
I
built
a
cloud
function
based
on
go
and
to
simulate
the
just-in-time
transformation
of
some
data
from
an
object,
store,
I
thought
hey.
A
This
would
be
sort
of
interesting
way
to
put
these
together,
so
I
built
it
out.
I
started
getting
data
and
unfortunately,
the
data
didn't
look
quite
that
cool
or
coherent.
What
I
saw
was
this
and
in
I
had
previously
thought
I
built
some
cross
cloud
systems
before
and
I
thought
you
know
going
to
cross
cloud
would
would
be
about
30%
slower.
A
What
I
actually
saw
was
there
was
about
more
than
twice
as
slow
in
some
cases
and
in
other
cases
it
was
four
times
slower
to
go
across
cloud,
and
so
so
I
backed
myself
into
a
corner
of
sorts,
because
I
choose
and
chosen
to
surplice
I
didn't
have
anyway
I've
seen
what
was
actually
going
on
with
the
network.
Normally
I
just
go
on
to
a
box
and
gonna
get
a
peak
app
and
start
looking
and
all
I
had
was
actually
be
the
application
interface
and
so
I
asked
myself.
A
How
can
I
see
what's
really
happening,
and
why,
and
so
I
did
some
research
about
different
ways.
You
could
do
this
and
go
and
I
eventually
found
the
package
HCP
trace
and
what
this
does
is.
It
gives
you
callbacks
for
a
start,
starting
in
time
for
DNS
TCP
TLS
and
the
actual
HTTP
request
in
response.
So
this
is
perfect.
This
had
the
data
I
needed
and
so
I
looked
for
a
way
to
integrate
it
into
the
traces,
because
I
wanted
to
see
not
not
in
aggregate.
Why?
A
For
me,
so
I
kept
looking
for
other
tools
that
were
using
this
and
I
found
that
the
AWS
x-ray
SDK
used
this
and
it
provided
a
really
great
interface
for
it,
but
it
was
for
x-ray
and
not
for
open
tracing,
but
luckily
open
source,
Apache
License
I
took
an
afternoon
ported
it
to
open
tracing
and
I
had
more
data.
The
data
looks
sort
of
like
this
DNS
style,
TLS
requests
in
response,
and
this
actually
made
sense
of
what
was
going
on
and
for
the
simplest
case
and
to
understand
why
it
makes
sense.
A
You
need
to
understand
this
diagram
here,
so
this
is
what's
involved
in
in
a
network
transaction
using
TLS,
so
you
have
syn
ack
ack
syn,
syn,
ack
ack
for
the
tcp
phase.
You
have
the
the
TLS
negotiation,
which
actually
has
way
more
messages
than
fit
on
this
screen.
But
you
have
to
back
and
forth
round
trips
for
that,
and
then
you
have
one
back
and
forth
for
the
HTTP
request.
I
put
the
storage
RPC
on
there's
well
because
that
actually
becomes
visible
shortly.
A
A
So
we
know
one
round
trip
in
this
case
is
76
milliseconds
and
we
say
two
round
trips
about
150
that
looks
about
right
and
then
we
look
at
this
and
we
say
well,
it's
one
round
trip
which
we
know
is
probably
about
75
milliseconds
at
this
point,
plus
25
milliseconds,
for
a
storage
RPC.
This
is
very
interesting,
so
you
can
you
can
start
tracking
these
different
sorts
of
parts
of
what's
affecting
your
latency
from
the
application
in
your
traces
and
then
so.
A
Looking
at
actual
cases
with
this
I
saw
interesting
things
like
this,
so
this
is
what
a
trace
for
cross
region
with
GCS
looks
like
and
you
you
look
at
the
network.
Latency
you're,
like
that,
looks
pretty
weird,
but
then
you
remember
that
google
has
local
endpoints
and
so
the
three
round,
trips
that
have
to
do
with
negotiate
connection
happen
locally,
and
then
only
this
large
storage
RPC
goes
cross
regionally.
A
Here's,
what
looks
like
us,
East
one
to
EU
west
one
and
even
that,
doesn't
really
convey
it.
It
wasn't
until
I
liked
I
was
trying
to
figure
out
how
to
illustrate
this
I
stuck
my
finger
on
the
globe
and
I
traced,
the
distance
I'm
like
yeah,
that's
a
really
long
ways,
and
so
that
that's
why
it
takes
a
long
time.
It's
just
just
the
physics
of
it.
So
they
explain
the
differences.
I
was
seeing
with
initial
setup,
but
the
big
difference
turned
out
to
be
this.
A
So
a
lot
of
the
time
I
was
serious
connection,
Rios
and,
and
so
one
round
trip
is
a
lot
less
than
three
round
trips,
especially
cross
regionally,
and
it
looked
like
this,
and
so
so
that
was
interesting,
but
the
the
problem.
The
confusion
was
that
it
happened
on
the
first
request
and
I
was
wondering.
Since
this
is
a
function,
it
has
life
cycle.
A
How
was
I
getting
reuse
on
the
first
request
and,
and
so
I
spent
a
lot
of
time
like
asking
questions
trying
to
figure
out
with
reflection
inside
of
the
thing
I
I
thought
my
hands.
I
said
it
must
be
run
time,
magic,
but
Google's
doing
in
Google
cloud
functions,
and
so
I
I
published
I
published
the
the
report
and
I
submitted
the
CFP
and
the
day
after
I
got
acceptance
for
this
talk.
I
got
this
or
named
lambda
and
I
learned
that
connections
are
not
magic.
What
I
saw
in
lambda
was
this.
A
Well,
if
it's
not
run
time
magic,
what
might
it
be
and
I
I
thought
about
all
the
different
dimensions:
I'd
taken
account
of
I'd,
taken
account
of
the
different
services
I'd
taken
account
of
the
region's
I'd,
taken
account
of
the
the
the
different
targets
for
the
different,
the
the
actual
storage,
but
it
turned
out
what
I
hadn't
taken
account
of
was
frequency,
so
I
had
been
making,
since
it
was
the
easiest
thing
to
do
a
bunch
of
requests.
Every
minute,
I
just
used
a
cron
job
and
made
the
requests.
A
It
turns
out
that
if
you
make
requests
more
often
you
get
reuse
with
s3
as
well.
So
this
is
this
is
the
what
the
histogram
looks
like
for
for
s3
and
you
can
see
as
you
get
more
frequent
requests.
You
pass
that
reused
threshold
and
you
start
not
having
to
pay
the
price
of
all
those
all
those
connections.
Setups
again
so
I
did
that
and
then
I
went
back
and
I
said
well.
A
It
was
still
seen,
reuse
and
at
that
point,
I
did
throw
up
my
hands
and
say
maybe
this
magic,
but
really
further
research
is
needed,
is
that
is
the
conclusion
I
came
to
so
I
saw
several
other
interesting
things
they'd
like
to
share
with
you
as
part
of
this
process,
so
the
first
one
by
default,
when
you
set
up
a
land
of
lambda
function,
it
will
assign
it
128,
megabytes
of
RAM
and
and
what
I
was
seeing.
Was
things
like
this?
So
using
the
same
rule
of
how
do
you?
A
A
Isn't
it
the
case
that
that
there's
supposed
to
be
native
support
for
TLS
operations
and
TLS,
so
in
most
CPUs?
Now
so
should
it
be
that
CPU
expensive
is
lambda
restricting
access
to
those
those
instructions
is
presenting
something
different.
Still
still
some
open
questions
there.
The
next
one
I
saw
was
GCSE
OAuth.
So
the
way
that
GCSE
works,
especially
outside
of
the
Google
cloud,
is
that
when
you
want
to
access
something
you
you
have
a
service
bundle
and
that
service
bundle
includes
a
key
and
a
certificate.
And
what
happens?
A
Is
the
SDK
uses
that
to
reach
out
to
an
OAuth
end
point
that
gives
you
back
a
token
that
then
you
can
use
in
your
actual
storage
in
point
and
and
there's
no
way
to
extract
that
client
to
see
how
long
it
takes.
So
you
you
end
up
with
this.
This
giant
amount
of
time
where
nothing's
actually
happening,
and
it's
a
long
time
on
average
it
could
be
one
round-trip.
It
could
be
two
round-trips.
It
could
be
full
set
up.
It's
not
visible
because
of
the
way
that
this
is.
A
This
is
actually
hidden
in
the
SDK.
So
what
I
ended
up
doing
is
there's
an
s3
Interop
mode,
that's
provided
in
GCS,
because
I
want
to
to
actually
present
the
best
possible
performance
here
and
with
that
with
that
you
wrap
mode
that
uses
HTML
key
signing
so
that
there's
no
back-and-forth
and
this
basically
goes
away.
It
also
goes
away
if
you
turn
off
off,
so
it's
definitely
OAuth,
but
why
is
it
so
slow?
That's
open,
open
question
next,
one
DNS
latency!
A
So
it's
actually
faster
to
ask
Google
for
a
DNS
answer
that
is
to
get
one
locally
seems
odd.
It
would
be
nice
to
understand
why,
but
without
knowing
that
that
you
can't
track
these
things
without
looking
at
you
just
be
like.
Oh,
it's
sort
of
slow
the
last
one.
Oh
the
last
one
I
when
you
run
the
Interop
mode.
A
What
I
saw
was
that
the
the
fetches
were
bimodal,
so
these
are.
These
are
two
different
two
different
peaks.
There's
nothing
different
in
the
dimensions
that
we
can
see
between
those
peaks,
so
they're
about
30
milliseconds
apart.
It's
a
little
bit
hard
to
see
with
the
log
notation,
but
they
all
have
connection
or
use.
They
all
are
to
the
same
same
region
and
they
all
have
the
same
qualifications
except
not
quite
half
of
them.
A
little
bit
less.
A
Is
it
going
to
be
latency,
but
it's
not
just
does
latency
matter
it's
like
does
latency
matter
from
what
locations
does
it
matter
at
what
request
rate
and
what's
the
acceptable
variance
of
latency
or
if
it's
throughput?
It's
with
what
parallelism,
with
what
matches
message
sizes
with
what
variance
again
and
when
you,
when
you
have
this
these
sorts
of
things
about
well.
This
is
the
kinds
of
things
I'm
trying
to
measure
in
the
system.
A
You
have
to
figure
out
what
perspective
you
want
to
measure
them
from,
because
it's
usually
not
enough
to
have
a
single
perspective.
It's
usually
not
enough
to
have
the
service
edge.
You
want!
You
want
other
perspectives,
so
you
can
make
sure
that
they're
oriented
towards
what
your
users
or
clients
or
whoever
else
is
you're
building
the
system
for
can
see,
and
you
you
have
to
start
start
thinking
about.
How
will
these?
How
will
these
perspectives
compare
with
each
other?
A
How
will
make
I
make
sense
of
them
and
that
that
next
step
is
in-
and
this
is
something
that
I
particularly
fail
to
do
explicitly
enough-
is
to
create
a
mental
model
of
what
you
think
the
moving
parts
of
the
system
are
going
to
be
and
how
you're
going
to
measure
that
and
how
you're
going
in
strim
--nt
so
that
you
can
confirm
or
disconfirm
your
mental
models
as
the
system
evolves.
As
you
have
more
users,
I
wish,
you
have
different
kinds
of
users.
A
How
I
know
that
that
what
I'm
thinking
what
I
would
I
believe
this
does
is
actually
what
it
does.
So
you
have
to
ask
what
am
I
saying-
and
this
is
one
of
those
basic
questions
where
you
think
you're
measuring
one
thing,
and
it
turns
out
that
you're
not
so
what
kinds
of
what
kinds
of
actions
can
you
take
to
try
and
try
and
see
if
what
you
think
you're
measuring,
isn't
what
you're
measuring,
if
you
think
you're
measuring
network
like
latency
for
connections?
What
can
you
do
to
see
that?
A
A
Why
am
I
seeing
it
what
what
sorts
of
things
might
be
causing
this
if
I
know
the
the
pieces
of
a
puzzle
that
I'm
trying
to
work
on
what
might
be
causing
it
to
do
something
different
than
what
I
expect,
but
the
most
important
lessons
for
this
for
me,
and
hopefully,
hopefully,
what
you
can
take
away
from
this-
is
that
various
happens,
but
it
happens
long
dimensions
and
you
might
not
know
what
those
dimensions
are,
but
various
is
not
random
like
these.
Are
these
are
fairly
deterministic
systems
even
than
their
complexity.
A
So,
if
you
see
a
large
amount
of
variance-
and
you
can't
understand
why
there's
probably
dimension
that
you're
missing
and
that
that
dimension
might
not
even
be
available
to
you,
but
you
you
have
to
have
to
think
about-
do
I
have
the
dimensions
to
understand
the
behavior
of
the
system.
What
sorts
of
things
would
I
need
to
understand
about
the
system
to
understand
if
the
variance
matters,
what
kinds
of
things
I
need
to
investigate
to
to
create
these?
A
The
other
important
part
of
this
is
that
it's
important
to
it's
important,
to
expose
to
your
users,
the
uncertainty
and
and
keep
that
fresh,
because
there's
there's
always
going
to
be
something
that
you
can't
see.
There's
always
going
to
be
something
that's
not
clear,
and
if
people
are
certain
that
things
mean
a
certain
way
that
the
measurement
absolutely
means
this,
then
they
can
make
decisions
that
can
that
can
take
your
your
system
off
the
rails.
So
so
keeping
that
uncertainty
fresh.
A
My
favorite
method
is
to
use
Chaos
X
exercises,
but
there
are
lots
of
different
ways
to
to
make
sure
that
you're
exposing
that
to
to
your
users
as
well
so
I'd
like
to
leave
you
with
with
with
an
exercise,
something
to
imagine
a
direction.
So
imagine
if
you
could
look
at
the
help
of
a
critical
cloud
service,
API
dependency
across
all
your
services,
the
same
way.
A
Imagine
if
you
could
have
the
various
that
matters
built
into
your
cloud.
Sdks.
Imagine
if
you
could
compare
the
healthier
scene
from
these
API
so
with
trusted
other
users
of
those
api's.
Instead
of
trying
to
back
channel
to
your
friends
and
slack
or
searching
Twitter,
imagine
real
Status
page
that
reflects
actual
use
along
the
dimensions
that
matter.
So
this
is
my
next
project
I'm
working
on.
How
do
we
collect
the
information
in
the
same
way
so
that
so
that
you
don't
have
to
ask?
Is
it
just
me
or
you
don't
have
to
ask?
A
Is
it
just
this
this
service?
How
can
we
take
the
opportunity
that
open
telemetry
provides
us
to
rethink
the
way,
we're
instrumenting
these
critical
dependencies,
and
if
you
want,
if
you
want
to
collaborate
with
us,
if
you
want
to
complete
it,
compete
with
us,
let's
figure
out
how
to
do
it.
Thank
you.
C
A
So
I
was
not
able
to
get
anybody
since
I
wasn't
paying
enough
money
to
actually
answer
questions
about
what
was
happening
with
the
interesting
opportunities.
I
think
exists,
though,
is
that
there
was
a
presentation.
I
recently
saw
about
how
AWS
does
their
failure
domains
by
cells,
and
so
that's
how
they
provide
the
personalized
dashboard.
It
would
be
really
interesting
if
they
provided
the
failure
domain
back
in
response
headers,
so
we
could
tag
our
stuff
up
that
way
and
if
Google
did
something
similar.
A
Basically,
it
seems
in
my
previous
experience,
dealing
with
support
teams
that
it's
it's
totally
decoupled
like
you,
you
send
them
a
graph
and
you
send
them
dimensions,
and
then
they
go
talk
to
some
other
people
and
they
pull
their
equivalent
graph.
But
it's
it's.
It
would
sure,
be
nice
to
be
able
to
to
have
even
on
demised
version
of
their
back-end
graph,
but
that
is
not
something
that
I
found
anybody
who's
interested
in
doing
yet,
unfortunately,.
E
Great
talk,
thank
you
very
much.
So
you
talked
a
little
bit
about
being
kind
of
blind
to
certain
things.
Underneath
the
hood
due
to
lack
of
instrumentation,
have
you
ever
used,
and
you
mentioned
using
chaos
engineering
a
little
bit
to
get
at
the
truth.
Can
you
talk
about
an
example
of
something
like
that
where
you
suspect
something's,
true
and
then
you
can
go
in
and
create
a
test?
That's
some
kind
of
gameday
experiment
that
validated
or
invalidated
your
kind
of
hypothesis,
yeah.
A
So
so
one
of
the
one
of
the
very
basic
things-
and
maybe
it's
too
basic
just
to
say,
but
one
of
the
things
when
one
of
the
systems
I
was
working
at
Twilio,
we
built
it
out
and
we'd,
we
thought
we'd
had
observability
and
that
what
we
start
doing
is
we
remove
capacity,
and
we
figured
out
that
we
weren't
actually
looking
at
any
of
the
in
10
metrics
from
a
business
centric
way
and
so
removing
capacity.
We
knew
we
moved
past
e.
We
couldn't
see
that
we'd
remove
capacity.
A
That
meant
that
we
weren't
actually
looking
at
the
things
that
mattered
in
another
case,
same
kind
of
thing,
where
we
just
started
injecting
Network
latency
to
to
once
of
the
endpoints
real
problem
that
can
happen.
Some
some
set
of
the
things
you
depend
on
gets
low,
absolutely
invisible,
because
we
weren't
we
weren't
separating
out
the
way
we
looked
at
at
the
at
the
metrics
and
traces.
A
way
that
would
would
allow
us
to
root
cause
that
kind
of
issue.
So
even
simple
things
like
removing
capacity
or
injecting
latency
can
can
surface
those
count,
gaps,
cool.