►
From YouTube: Migrating to OpenTelemetry From a Custom Distributed Tracing Pipeline - Francis Bogsanyi, Shopify
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Migrating to OpenTelemetry From a Custom Distributed Tracing Pipeline - Francis Bogsanyi, Shopify
Shopify built its own distributed tracing pipeline in 2016, including custom instrumentation, a custom propagation format, trace collection, down sampling, augmentation, cleansing and fanout to multiple analytics backends. Over the past year, they have been migrating their entire tracing pipeline to OpenTelemetry, after a brief sojourn with OpenCensus. This talk describes the motivation for the migration, the advantages of working with and building upon the OpenTelemetry project, and concrete details of the migration process.
https://sched.co/ZekF
A
Hi,
I'm
francis
I
work
on
distributed
tracing
at
shopify
and
I'm
also
a
maintainer
of
open
telemetry
ruby
in
2016
shopify
built
its
own
distributed
tracing
pipeline
last
year
we
started
migrating
our
entire
tracing
pipeline
to
open
telemetry.
This
talk
provides
a
historical
background
for
that
move.
Why
we
chose
to
build
upon
open,
telemetry
our
migration
strategy
and
what's
gone
right
and
wrong.
So
far,.
A
There's
four
major
things:
we
need
to
implement
distributed
tracing.
Firstly,
we
need
to
instrument
our
own
services
at
shopify.
Most
of
our
services
are
rails
applications,
so
we
needed
instrumentation
in
ruby.
Secondly,
all
the
services
need
to
agree
on
how
to
propagate
trace
context
between
themselves.
A
A
When
we
started
there
wasn't
much
publicly
available
for
ruby,
the
zipking
gem
had
been
abandoned
and
development
had
shifted
to
open
zipkin,
but
it
was
pretty
early
days.
Jaeger
hadn't
been
open
sourced
yet
and
it
lacked
instrumentation
for
ruby.
Anyway,
open
tracing
didn't
exist
for
ruby
and
when
it
showed
up
later,
the
api
didn't
feel
especially
idiomatic
to
us.
A
More
generally,
instrumentation
tended
to
be
tightly
coupled
with
the
storage
back
end.
While
some
commercial
tracing
back-ends
existed,
there
wasn't
a
clear
winner
and
they
all
provided
their
own
instrumentation
libraries.
Given
this
landscape,
we
did
the
obvious
thing
and
wrote
our
own
instrumentation
library.
A
The
story
for
context
propagation
was
similar.
The
headers
tended
to
be
specific
to
the
instrumentation
b3
multi-header
from
zipkin
and
x.
Cloud
trace.
Context
from
stackdriver
were
the
two
obvious
choices
we
didn't
want
to
pay
for
a
back
end.
While
we
were
building
out
this
functionality,
stackdriver
trace
and
datadog
trays
were
free
at
the
time
with
some
limitations.
A
A
We
ended
up
building
the
custom,
instrumentation
library
inspired
by
open
tracing
it
had
auto
instrumentation
for
all
the
things
we
commonly
use
at
shopify,
elasticsearch,
redis,
memcache,
rails,
mysql
and
so
forth.
We
had
just
moved
into
gcp
and
it
seemed
like
a
good
idea
to
leverage
gclb's
tracing
support,
so
we
used
x,
cloudtrace
context
for
propagation.
A
A
A
A
Firstly,
we
had
a
bottleneck
that
allowed
us
to
massage
spans
in
different
ways
like
redacting
pii,
cleaning,
up
span
names
and
remapping
tag
keys
to
things
that
the
backend
understood
and
treated
specially.
It
also
let
us
easily
migrate
between
versions
of
the
backend
api,
without
interfering
with
the
services
that
were
using
our
instrumentation.
A
Secondly,
we
could
experiment
with
different
back
ends,
an
experiment
we
did
by
building
additional
translators
and
adding
members
different
subscribers.
We
could
easily
bake
off
different
back
ends
with
the
same
data.
I
think
we
had
five
back
ends
running
simultaneously.
At
one
point,
we
tried
zipkin
jaeger,
new
relics,
tracing
service,
signalfx,
lightstep
and
omniscion,
alongside
stackdriver.
A
Although
we're
known
as
a
ruby
shop,
we
have
quite
a
bit
of
go
some
rust,
some
nodes,
some
python,
some
java,
we
built
a
go
tracing
package
early
on
that
was
compatible
with
open
tracing,
but
that
was
it
for
cross
language
support
and
every
time
we
wanted
to
try
out
a
new
backend.
We
had
to
write
custom
translators
and
learn
about
new
and
interesting
limitations,
as
we
often
directed
a
fire
hose
of
spans
at
a
pool
of
proprietary
collectors
or
satellites.
A
A
We
weren't
the
only
ones
to
realize
this,
where
open
tracing
spectrum,
api
and
semantic
conventions,
google
built
a
set
of
batteries,
included
instrumentation
libraries
for
a
bunch
of
languages
with
plugable
exporters
called
census.
They
open-sourced
this
as
open
census
and
some
other
folks
got
on
board,
including
omniscient,
who
built
a
collector
for
the
project.
A
This
was
just
what
we
needed.
Ruby
isn't
exactly
google's
thing
so
that
library
was
in
rough
shape,
but
the
javascript
instrumentation
filled
a
gap
for
us
and
the
go
instrumentation
was
much
better
than
what
we'd
built
we
started
with,
go
and
built
a
simple
exporter
and
context
propagation
to
make
it
work
in
our
environment.
We
then
turned
our
attention
to
the
collector.
A
A
A
A
We
then
needed
to
shift
traffic
to
the
new
pipeline.
We
set
up
two
export
pipelines
in
the
collectors
one
to
stackdriver
and
one
to
omniscient.
We
added
a
reverse
proxy
to
the
trace
proxy
that
simply
forwarded
requests
to
the
upstream
collectors,
and
we
added
a
mechanism
in
the
trace
proxy
that
let
us
dynamically
control.
The
traffic
slid
by
adjusting
the
config
map
that
was
mounted
as
a
file.
Config
trace
proxy
proxy
percentage.
A
A
In
both
cases,
we
only
needed
to
implement
context,
propagation
and
the
collector
solved
our
third
problem,
maintaining
custom
translators
for
every
backend.
We
only
needed
to
implement
a
single
translator
from
our
custom
format
to
the
collectors
in
memory
format.
Ultimately,
we
wanted
to
replace
that
custom
format
as
well
and
our
custom
instrumentation
in
ruby.
So
we
decided
to
take
on
the
challenge
of
maintaining
open
sensors
ruby.
A
Now
we're
into
the
meat
of
our
journey
to
open
telemetry
halfway
into
our
migration
to
open
census.
We
had
to
make
this
shift.
It
was
awkward
to
continue
with
the
opencensus
collector
because
it
had
been
placed
into
maintenance
mode,
as
code
was
copied
over
to
the
open
telemetry
project
and
all
development
effort
quickly
shifted
there.
A
A
We
deployed
the
collector
into
the
same
name
space
and
then
pointed
the
kubernetes
service
of
the
open
telemetry
collector,
instead
of
the
open
census
collector.
Obviously
we
did
this,
starting
with
less
important
clusters,
gave
it
some
soak
time
and
then
gradually
migrated
more
to
more
and
more
important
clusters.
A
A
The
agent
is
really
just
a
collector
with
a
smaller
footprint
and
a
slim
down
pipeline.
If
you
appear
closely
you'll
see
an
additional
processor,
we
built
here
called
resource
labeler.
This
grabs
metadata
about
the
node
and
its
environment
from
a
couple
of
different
places
and
augments
spans
with
that
information
things
like
the
node
name,
the
cluster
name,
gke
version
availability
zone.
A
In
the
meantime,
a
team
that
owned
the
shopify
scripps
runtime
engine
wanted
to
add
tracing
the
runtime
engine
was
written
in
rust
and,
coincidentally,
the
open
telemetry
russ
sig
had
just
released
an
alpha
of
the
sdk
in
rust,
so
with
minimal
handholding
for
us
and
an
awesome,
can-do
attitude
from
them.
We
shopify
a
hardcore
rail
shop,
shipped
open
telemetry
to
production
for
the
first
time
in
rust.
A
The
most
recent
thing
we've
built
is
an
otlp
exporter
from
our
customer
instrumentation
library.
Otlp
is
open.
Telemetry's
native
protocol
for
telemetry
data
in
large
part.
It's
a
refinement
of
the
open
census
protocol,
but
it's
a
huge
step
forward
in
computational
efficiency,
and
it
is
also
a
lot
friendlier
to
load
balances.
A
Our
journey
hasn't
been
all
sunshine
and
roses.
Previously
I
glossed
over
the
stress
and
anxiety
of
switching
from
the
open
senses
collector
to
the
open,
telemetry
collector.
Much
of
that
was
unwarranted
and
the
transition
was
actually
pretty
smooth.
The
agent
has
given
us
far
more
grief,
though
we
wanted
the
agent
to
listen
on
the
same
port
as
a
trace
proxy
for
our
custom
protocol.
So
the
initial
deployment
was
nerve-racking.
A
We
relied
on
custom,
we
relied
on
client
applications,
buffering
spans
and
reconnecting
to
cope
with
brief
outages
on
each
node.
As
we
deployed
the
agent
and
removed
the
trace
proxy
demon
set,
deploys
in
our
environment,
terrible
for
everyone
and
rolling
out
a
change
across
all
clusters
often
takes
a
day
and
a
lot
of
hair
loss,
covert.
Hair
helps
me
hide
some
of
that.
A
While
the
trace
proxy
used
well
under
128
mega
ram,
we
needed
2
gig
for
the
agent
given
how
tightly
we
pack
our
nodes.
We
couldn't
afford
to
give
it
more
cpu
than
the
trace
proxy
had,
but
by
golly
it
needs
more.
Cpu
throttling
in
the
agent
has
been
a
major
pain
point.
The
crux
of
the
problem
is
that
it
needs
to
deserialize
from
the
receiver's
format
copy
and
translate
to
the
in-memory
format
and
then
copy
and
translate
to
the
exporters
format
before
re-serializing.
A
Those
improvements
come
at
the
cost
of
breaking
changes
in
the
collector.
It
has
been
a
struggle
to
keep
pace
with
internal
api
changes
in
our
custom
receiver.
Until
recently,
there
was
also
some
churn
in
metrics,
which
increased
stress
and
anxiety
when
rolling
out
upgrades
an
otlp,
while
amazing
has
experienced
churn
of
its
own.
We
successfully
advocated
for
declaring
the
trace
portion
of
the
protocol
is
stable,
but
in
the
meantime
we
had
to
deal
with
chune
in
the
spec
and
mismatches
between
the
upstream
proto
definitions
and
the
version
vended
by
the
collector.
A
A
Once
we
migrate
all
our
services
to
export
in
otlp,
we
can
start
deleting
more
custom
code
and
eventually
consider
building
directly
from
the
upstream
repo
open
telemetry
ruby
is
used
by
some
companies
in
production.
We
need
to
curate
it
a
little
for
our
environment,
but
we're
close
to
trialing
it
in
a
few
services.
A
The
last
big
hurdle
we
have
to
fully
migrate
to
open
telemetry
is
replacing
our
context
propagation
with
the
w3c
trace
parent
format.
This
is
a
default
for
open
telemetry
and
should
see
increasing
adoption
across
many
third-party
applications
more
specific
to
our
environment.
We
have
a
lot
of
clusters
across
fewer
regions
and
rolling
our
code
or
configuration
changes
for
collectors
across
all
of
them
is
a
maintenance
headache,
even
without
the
agent
in
particular.
A
Some
clusters
are
pretty
small
and
don't
auto
scale
for
reasons,
so
we
invariably
have
to
babysit
deploys
in
clusters
that
our
team
doesn't
own,
there's
also
a
lot
of
variability
in
traffic
across
these
clusters,
so
one
size
really
doesn't
fit
all.
Instead,
we're
building
out
regional
pools
of
collectors
to
replace
the
cluster
local
pools.
A
A
The
first
lesson
is
that
a
migration
like
this
takes
a
long
time,
particularly
in
our
case,
we
had
a
very
small
team
and
open
telemetry
was
new
and
changing
rapidly.
Some
of
that
is
likely
easier
now,
but
not
all
of
it.
The
second
lesson
is
that
working
backwards
from
the
end
of
the
pipeline
worked
really
well.
We
could
simply
extend
our
pipeline
to
use
a
collector,
then,
once
we
were
confident
we
introduced
it
into
production
clusters,
then
gradually
switched
the
pipeline
over
to
use.
It
then
removed
the
old
pipeline.
A
A
Finally,
trace
instrumentation
and
collection
is
commoditized,
there's
no
value
in
building
or
maintaining
this
yourself.
The
real
value
is
at
the
end
of
the
pipeline
and
analysis
and
monitoring
and
alerting
there's
a
lot
of
innovation
and
churn
and
consolidation
in
this
space.
It's
good
to
head
your
bets,
open
telemetry,
helps
you
do
that.