►
Description
eBay is a high scale end user of open source observability. Collecting telemetry from millions of endpoints, and running thousands of queries per second, presents serious scaling challenges. eBay has chosen to use an open source stack to meet those challenges.
On this episode I hosted Vijay Samuel, Observability Architect at eBay, to hear about the challenges eBay faced in monitoring large Kubernetes installations. We discussed why eBay chose the open source stack, and the strategic decision behind eBay's migration from Elastic Beats to OpenTelemetry, and the remarkable experiences they had while deploying massively large scale telemetry installations using OpenTelemetry and Prometheus.
We also discussed a fascinating new proposal for a query language standard for observability, which eBay spearheads together with Netflix under the auspices of the CNCF’s TAG Observability (the CNCF’s technical advisory group for observability).
Vijay Samuel works with eBay's observability platform as its architect. During his time at eBay Vijay has transformed eBay's observability platform into a cloud native offering that is primarily built on top of open source technologies. He loves to code in Go and play video games.
The podcast episodes are available for listening on your favorite podcast app and on this YouTube channel.
We live-stream the episodes, and you’re welcome to join the stream here on YouTube Live or at https://www.twitch.tv/openobservability.
Follow us on Twitter @openobserv to get the live stream times and other updates, and to pitch in with your thoughts and comments.
Dotan Horovits
============
Twitter: https://twitter.com/horovits
LinkedIn: https://www.linkedin.com/in/horovits/
Vijay Samuel , eBay
===============
Twitter: @vjsamuel_
LinkedIn: vjsamuel
Resources
=========
New CNCF working group for Observability Query Language Standardization: https://github.com/cncf/tag-observability/blob/main/working-groups/query-standardization.md
Why and How eBay Pivoted to OpenTelemetry: https://tech.ebayinc.com/engineering/why-and-how-ebay-pivoted-to-opentelemetry/
Chapters
========
00:00 - show intro
00:33 - 3rd anniversary
02:00 - episode intro
04:48 - eBay monitoring solution
07:38 - planet scale at eBay
09:57 - distributed tracing at eBay
16:28 - migrating from Elastic Beats to OpenTelemetry
19:36 - why eBay chose open source
22:47 - open-sourcing eBay’s metrics store platform?
25:19 - scaling Prometheus
35:12 - ramping up tracing, from backend to frontend and mobile
40:03 - running OpenTelemetry Collector at scale
49:05 - Query Language Standardization for Observability under the CNCF
56:39 - Vijay Samuel contact and final notes
59:12 - episode outro
A
Hello,
everyone
and
welcome
to
open
observability
talks,
I'm
Jordan,
horvitz
and
here
at
open,
observability
talks.
We
talk
about
anything,
devops,
observability
and
open
source,
so
May
the
open
source
be
with
you
and
I'm
glad
to
say:
we've
just
concluded
the
third
season
of
the
show
the
third
year
going.
That's
an
amazing
achievement
and
I
want
to
thank
you
all
for
the
feedback,
for
the
support
for
following
I
actually
wrote
a
summary
post
for
the
year,
so
I'll
post.
It
also
here
for
the
for
the
members.
A
You
can
check
it
out
here
and
obviously
for
the
listeners,
that's
on
the
live
stream
and
for
the
listeners
on
the
podcast
I'll,
also
post
it
on
the
show
notes-
and
we've
done
so
many
so
much
in
this
last
year,
I'm
amazed
to
see
we
touched
much
broader
than
just
operational
observability.
We
touched
finops,
we
touched
platform
engineering,
we
touched
so
many
end
user,
like
meta
and
others,
and
also
the
projects,
obviously
open,
Telemetry
and
Prometheus
and
so
on.
A
So
that
was
an
exciting
year
and
now
I'd
like
to
also
thank
our
sponsors.
Logs.Io.
The
client
native
observability
platform
logs
IO,
takes
the
best
of
read
open
source
projects
such
as
Prometheus,
open,
search,
Jager
and
open
Telemetry
and
offers
them
as
a
unified
observability
platform
built
for
scale
for
those
joining
the
live
stream
on
YouTube
or
twitch
feel
free
to
share
questions
and
comments
on
the
live
chat.
A
A
Yeah,
it's
amazing
and
I.
Just
realized
I
told
you
that,
just
before
the
the
broadcast
that
I
realized
the
first
time
we
met
in
person
was
last
year
at
qco,
North
America
in
Detroit,
with
the
fellows
at
the
dinner
there
for
tag
observability,
the
technical,
Advisory
Group
and
that's
like
a
a
closure
now
that
you're
a
part
of
the
tag,
observability
yeah,
with
the
new
initiative
with
the
with
the
query
language
standardization.
So
that's
that's
exciting
to
get
you
on
the
show.
B
A
C
B
The
the
there
are
many
cool
things
that
come
out
of
it
and
like
a
fun
fact,
is
that
the
SRE
can't
talk
and
scale
talk
that
we
did
about
our
planet
scale.
Prometheus
installation
was
actually
vetted
out
and
approved
during
one
of
the
past
hack
hack
weeks
or
invent
week.
So
the
the
timing
is
good.
So,
given.
C
C
B
C
B
A
An
absolute
pleasure
and
by
the
way
I
I
had
on
on
this
show
one
of
the
episodes
was
with
Ben
siegelman
who's
being
part
of
the
team
at
Google
who
implemented
and
supported
and
scaled
out
Monarch
within
Google.
So
it
was
a
fascinating
episode
for
those
who
look
for
this
episode:
the
expensive
observability.
That
was
the
name
of
the
episode,
and
he
really
shared
some
very
cool
insights
about
the
journey
that
Google
took
internally
to
come
up
with
Monarch
and
for
those
who
don't
know.
A
Monarch
is
the
internal
implementation,
the
internal
tooling,
that
Google
created
for
their
metrics
store
and
analytics
so
a
fascinating
episode,
and
in
general,
what
you
said
about
the
hack
week
even
last
month
when
I
had
the
episode
with
spotify's
senior
engineering
manager
about
how
backstage
went
open
source.
It
also
started
with
hack,
quick,
Spotify
hack,
quick,
where
they
proved
that
it's
it's
feasible
to
take
the
existing
project
and
and
open
source
it
and
so
yeah.
A
The
power
of
hackathons
to
their
variants
is,
is
amazing
and
it's
glad
to
say
that
it's
it's
live
in
kicking
also
in
large
organizations
such
as
eBay,
and
you
mentioned
the
scale
and
I
saw
your
talk
at
cubecon
if
I'm,
not
mistaken,
probably
or
maybe
another,
but
I
was
struck
by
the
the
scale
of
your
operation.
Can
you
give
us
some
sense
of
the
scale
of
observability
for
eBay.
B
Sure,
yeah,
with
regards
to
metrics,
at
least
at
any
given
point
in
time.
We
roughly
have
like
two
and
a
half
million
Prometheus
and
points
that
are
being
script,
which
translates
to
roughly
60
million
actor
60
million
samples
per
second
that
we
ingest
into
the
platform
and
roughly
two
and
a
half
billion
active
series
that
we
store
on
the
Prometheus
head
blocks,
and
these
series
are
pretty
much
helping
us
do
a
detection
on
top
of
business
metrics.
B
If
the
core
indicators
are
through
anomaly
detection,
we
know.
If
they
are,
they
are
misbehaving,
then
we
have
health,
aware
rollouts
using
kpis
and
some
anomaly
detection
models
as
well
to
figure
out
if
there
is
a
bad
change
being
rolled
out
or
not
so
that
we
can
trigger
a
quick
roll
back
and
like
all
these,
basically
are
just
querying
I
shouldn't,
say:
good
English,
they're
bombarding
the
system
with
at
least
10
000
queries
per
second
and
at
any
given
point
in
time.
B
So
like
we,
we
also
have
a
RSI
rsre
Partners
have
also
built
slos
on
top
of
the
the
data
and
burn
rate
alerts
and
whatnot,
that's
helping
them
know
how
all
the
apis
are
behaving
and
whatnot.
So
there
are
like
a
lot
of
complex
use
cases
that
we
have
built.
We
also
have
a
a
huge
logs
platform:
that's
there,
which
pumps
in
roughly
15
petabytes
of
logs
every
day.
B
So
it's
a
it's
a
it's
a
pretty
massive
scale
shop
that
we
run
and
that's
one
of
the
things
that
I
really
enjoy
working
at
eBay,
because
you,
you
don't
get
to
play
with
such
scale
in
in
a
lot
of
organizations,
and
the
learnings
have
been
pretty
good
for
our
team
as
well,
which
forces
us
to
push
the
boundaries
and
the
planet
scale.
Metrics
platform
is,
is
a
very
good
example
of
that.
We.
B
Racing
now
trying
to
deliver
instrumentation
through
our
managed
Frameworks
into
all
applications
that
part
of
the
eBay
site,
so
that
we
can
also
have
a
better
time
to
triage
than
what
we
have
right
now.
Still
very
early
early
beta
I
should
say.
But
hopefully
we
should
have
like
a
pretty
sizable
tracing
installation
in
the
coming
months
or
maybe
in
next
year.
A
C
B
We
use
the
tracing
sdks,
we
have
applications
that
instrument
primarily
with
the
Java
and
node.js,
and
also
some
applications
in
in
go
as
well
on
every
kubernetes
cluster
that
we
run.
We
have
open,
Telemetry,
collector
running
local
to
the
cluster,
so
that
applications
knowing
a
well-known
endpoint
within
the
cluster
they
can
ship
their
spans.
B
We
do
enrichment
generated
metrics
and
we
all
where
then
we
ship
it
into
our
backend,
for
which
we
use
clickhouse
right
now,
and
we
also
have
exemplars
enabled
on
all
the
free
instrumentation
that
we
ship
through
our
managed
Frameworks,
so
that
specifically
things
like
latencies,
we
have
exemplars
as
well.
So
what
we
the
best,
the
three
click
RCA,
where
you
start
from
an
alert
you
go
to
a
graph.
B
You
see
an
example
or
you
click,
it
look
at
the
waterfall
and
then
from
there
you
have
an
a
clickable
attribute
for
the
logs.
You
click
there
and
then
you
look
at
the
raw
logs.
So
that
is
the
experience
that
we
are
trying
to
Champion
so
that,
like
people's
time
to
triage,
gets
cut
down,
we
are
working
through
it
because,
like
exemplars,
are
still
like
very
nascent
technology
right
now.
B
Even
if
you
take
Prometheus
as
an
example,
they
only
have
in-memory
exemplars,
it
doesn't
support,
recording
rules
which
I
have
a
PR
out
and
we
are
working
through
the
reviews
and
whatnot,
but
yeah
at
least
the
feeling
that
we
have
internally
is
that
an
exemplar-based
approach
is
very
powerful.
It
doesn't
require
you
to
store
100
samples
and
then
do
tail
sampling
on
top
of
that,
and
things
like.
So
there
are
a
lot
of
benefits
which
we
are
trying
to
explore.
Tracing
is
ever
revolving
I.
B
Try
to
tell
people
as
well
like
it's
it's
pretty
hard.
We
have
done
so
many
iterations
of
collector
pipelines,
one
tier
two
tier
back
to
one
tier
and
then
two
tier
again
like
to
get
things
figured
out
in
a
in
a
cost
optimized
way.
While
we
provide
enough
business
value
for
our
internal
developers
to
actually
use
the
product,
so
a.
A
Lot
of
learning
I
see
it
in
many
organizations
that
trying
to
get
to
Champion
distributed
tracing
within
the
organization
has
some
resistance,
or
at
least
not
clear,
for
everyone.
What's
the
benefit
and
the
investment
is
is
significant.
You
need
to
instrument
across
the
organization
to
really
get
the
value
you
need
to
have
all
the
the
parts
of
the
chain
instrumentals.
You
can
actually
see
the
the
transaction
or
the
request
going
flowing
through
the
and,
what's
the
ROI,
that's
always
the
question
that
I
keep
on
encountering
and
for
us
it's
it's.
B
B
There
have
been
wins,
but
if
we,
if
we
expect
every
application
owner
within
the
company
to
do
manual,
instrumentation
it's
a
very
hard
problem
and
yeah
I,
don't
think
the
engineering
effort
would
be
would
return
on
the
investment.
So
that
is
where,
like
we
have
picked
an
approach
where
we
instrument
into
the
framework
that
everyone
uses-
and
this
framework
keeps
getting
upgraded
every
month
through
a
mandatory
site
wide
upgrade.
B
So
it's
very
easy
for
us
to
roll
out
new
instrumentation
and
we
say
like
try
not
to
instrument
on
your
own
as
well
because
like
if
you
have
like
a
very
big
call
chain,
and
then
you
have
like
custom
spans
on
top
of
it,
you
are
basically
being
a
Noisy
Neighbor
for
everyone
on
the
Chain
if
they
are
trying
to
debug
something
so
tracing
like
I.
I
cannot
stress
it
enough,
like
it's
a
very
hard
problem
that
is
very
easy
to
screw
up
as
well,
so
yeah.
A
I
think
that's
that's
a
great
way
to
go
in
large
organizations
and
also
to
reduce
the
noise
and
effort
for
the
individual
teams,
because
it
definitely
not
keen
to
do
the
instrumentation
themselves,
so
I'm
sure
that
they're
they're
not
complaining
about
that.
But
what
I
wanted
to
talk
about?
Really.
You
mentioned
several
open
sources.
A
You
mentioned
click
house,
and
you
mentioned
open
Telemetry,
and
you
mentioned
others
and
Prometheus,
and
and
that's
that's
a
key
thing
that
I
see
and
not
taken
for
granted
for
an
organization
such
as
yourselves
to
use
open
source
extensively
and
and
I.
Think
your
like
power
users
back
in
the
day,
even
with
with
elastic,
beats
right
for
for
collecting
logs
and
metrics
file
bit
metric,
beat
I.
Think
you
probably
want
to
still
one
of
the
biggest
most
prominent
contributors
to
the
to
the
Beats
open
source
projects
right.
B
Some
of
the
features
that
we
wanted
were
readily
available
on
on
open
Telemetry
as
well
like
specific
for
metrics.
A
good
example
would
be
that
stale
markers
the
Prometheus
equipment
manager
is
being
used
inside
of
the
open,
Telemetry
collector
and
it
emits
tail
markers
correctly,
and
that
was
a
problem
that
we
had
with
metric
Beach.
B
And
if
you
were
to
implement
steel
markers
on
top
of
it,
it
would
double
the
memory
cost,
because
you
need
to
store
the
previous
sample
and
then
do
a
steel
marker
on
top
of
that
and
it
would.
It
would
be
something
that
the
community
needs
to
accept
as
well.
So
there
are
like
many
reasons
for
us
and
tracing.
We
started
off
with
open
Telemetry
as
well,
which
would
mean
that
we'd
be
half
on
one
side
and
half
on
the
other
side
so
like.
B
Given
that
racing
was
there,
we
mindfully
moved
the
metrics
and
we'll
eventually
move
logs
into
open
Telemetry
as
well,
given
that
more
or
less
it's,
it's
the
it's
the
standard
that
everyone
will
have
to
align.
So
that
being
said
like
yes,
at
one
point,
we
were
one
of
the
biggest
installations
of
Beats
that
exists
and
now
I'm
I'm,
hoping
that
we
are
one
of
the
bigger
installations
of
open,
Telemetry
collectors
that
that
exist
in
the
in
the
wild.
B
A
So,
let's
talk
about
this
like
very
basic
thing
before
we
even
go
down
to
the
technicalities,
why,
in
an
organization
and
a
vendor
such
as
like
a
company
such
as
yourselves,
why
go
the
open
source
path
in
in
the
first
place?
Why
not
go
with
a
vendor
that
will
offer
you?
Maybe
even
all
the
these
types
of
functionality
and
Telemetry
types
and
whatnot
as
part
of
one
Enterprise
closed
Source
solution.
B
The
the
first
reason
is
the
community.
Obviously,
there
are
like
plethora
of
people
coming
from
different
kinds
of
organizations
who
are
looking
at
different
problems,
and
then
they
are
trying
to
solve
it
in
the
in
the
open.
It
provides
us
an
opportunity
to
be
a
good
citizen
and
then
contribute
new
features
like
a
good
example
on
open
Telemetry
would
be
that
it
did
not
have
support
for
Exemplar,
scraping
on
the
Prometheus
receiver
and
one
of
my
teammates
Aishwarya.
B
So
I
think
we
run
at
at
a
scale
where
it's
hard
for
us
to
build
everything
ourselves,
but
at
the
same
time
it's
not
easy
for
us
to
go
and
buy
something
and
expect
it
to
break
even
in
terms
of
cost.
So
so
the
happy
middle
part
is
to
be
a
good
citizen
on
the
open
source.
Communities
show
what
problems
that
we
see
at
scale,
which
a
lot
a
lot
of
these
project
contributors
would
probably
not
see
and
also
contribute
to
make
it
better.
We
did
it
for
Beats.
B
We
did
a
lot
of
bug,
fixes
scale,
optimizations
and
also
features
like
if
you
have
seen
hints
based
Auto
discover
to
use
annotations
on
your
pod.
Spec
to
tail
log,
fails
or
scrape
endpoints
and
whatnot.
That
is
something
we
basically
worked
with
them
and
then
got
it
to
a
point
where
it
is
a.
It
is
a
proper
feature
for
for
Beats
to
consume,
and
it
is
one
of
the
more
feature
Rich
annotation
based
kubernetes
Discovery
mechanisms
that
are
out
there.
B
So
when,
when
we
work
together,
better
things
can
come
out
and
when
I
wrote
the
first
feature
internally,
it
was
nothing
compared
to
what
we
came
up
with
when
we
worked
with
the
community.
It
was
a
far
better
product,
so
I
think
that
is
one
of
the
big
part
enablers
in
open
source,
where
we
always
think
that
we
we
have
the
best
best
implementation
but
like
when
you
work
with
other
people
who
potentially
are
smarter
than
you.
You
come
up
with
the
more
graceful
implementations
that
can
help
a
lot
more
people.
A
Yeah,
that's
for
sure,
and
you
talked
about
your
major
contributions
to
established
open
source
such
as
elastic
beats
back
in
the
day
that
there
was
a
it,
was
an
Apache
to
ecosystem
and
and
open
Telemetry
now
and
so
on
and
and
they're
also
your
own
projects
that
you
build
from
scratch
that
you're,
maybe
considering
open
source.
Do
you
want
to
say
word
about
that.
B
Yeah
so
like
we
are,
we
are
passively
exploring
what
it
would
mean
to,
or
is
it
if
people
would
be
interested
in
in
consuming
something
like
our
metric
store
platform
or
not
consuming
contributing
is
the
right
the
right
word
to
to
use.
So
if
anyone
thinks
that
what
we
have
built
is
cool
and
it
would
be
nice
if
it's
out
in
the
open
for
people
to
come
in
and
contribute
and
at
the
same
time
add
value
to
their
organization
from
an
observative
perspective.
B
Please
do
reach
out
to
us
and
to
to
go
back
to
the
backstage
example
like
this
this
this
this
year,
at
least
in
invent.
We
are
actually
looking
at
okay.
Can
our
stuff
be
deployed
in
Google
cloud
or
any
public
cloud
provider
because
like
if
we
decide
to
open
source
one
of
the
things
that
needs
to
work
is
it
needs
to
properly
work
on
on
other
Cloud
providers,
because
we
we
run
our
own
cloud
internally,
so
have
happy
to
hear
feedback
and
interest.
A
Yeah,
so
that's
you
heard
it
here
first
and
on
open
observability
talks,
folks,
there's
very
exciting
and
and
very
impressive
implementation
of
metric
store
and
analytics
back
end
at
scale
that
we'll
hear
a
bit
more.
But
if
you
are
interested
and
let's
encourage
eBay
to
contribute
that
to
the
open
source
and
show
your
enthusiasm,
I
mean
taking
part
in
contributing
and
getting
involved
in
this
project,
and
we
did
to
help
move
the
needle
and
get
it
to
that
to
be
done.
A
So
Vijay
will
mention
also
the
how
to
contact
Vijay
at
the
end
of
the
episode
and
you'll
have
that
also
on
the
show
notes
for
the
on
the
podcast,
but
definitely
a
great
call
out
and
thanks
for
sharing
it
with
us
and
I'm
sure
our
audience
are
the
perfect
audience
for
this.
So
you
came
to
the
right
place
and
let's,
let's
talk
about
about
the
details
that
impressive
talk,
that
I
heard
about
scaling,
Prometheus
and
the
planet
scale.
Observability
then
take
us
through
this.
C
B
The
the
one
of
the
problems
that
we
faced
initially
when
we
did
the
first
pass
of
our
metric
store
implementation
was
that
we
had
the
concept
of
clusters.
A
cluster
can
have
X
number
of
shards,
which
means
that
you
can
take
a
Time
series.
You
can,
you
can
hash
mod,
it
figure
out
which
Shard
it
means
to
land
on.
So
instead
of
having
like
a
behemoth,
one
one
terabyte
memory
worth
of
single
Prometheus,
you
could
essentially
say
that
I
have
like
20
shards
of
size,
50
gigabytes
per
Shard
and
whatnot.
B
So
we
we
went
through
that
exercise
and
then
we
figured
out
that
okay,
it's
probably
good-
to
have
a
Shard
that
accommodates.
Maybe
20
million
active
time
series
on
on
the
head
block
and
20
million
is
an
arbitrary
number
that
I'm
sorry.
So
what
if
a
use
case
comes
in
that
has
say
600
million
or
800
million
in
cardinality,
we
have
certain
use
cases
that
that
are
at
I
think
300
million
per
internal
tenant.
B
We
like
to
call
them
namespaces,
so
what
happens
then
Prometheus
will
absorb
all
of
it
without
any
problems.
It
will
store
it
in
the
head.
Blog,
the
more
memory
you
give
it
it's
fine,
but
the
real
problem
comes
in
when
you're,
trying
to
query
it
out
and
and
you're
trying
to
make
me
a
good
meaning
out
of
it
in
terms
of
either
anomaly,
detection
or
whatnot.
B
Okay,
this
is
the
query
that
you
are
looking
for,
or
the
series
that
you're
looking
for
it's
there
in
region,
one
region
two,
but
not
there-
in
Region,
Three,
so
I,
Should,
Skip,
all
the
shards
in
Region
Three
and
then
just
query
where
the
data
is
alone
available.
So
that
is
the
the
path
that
at
least
Monarch
prescribed
in
the
white
paper
and
they
do
it
using
what
is
called
a
field
hint
index.
B
So
every
label
the
metric
name,
you
either
generate
engrams
or
you
do
a
you
generate
a
fingerprint
out
of
it,
and
then
you
store
the
fingerprint
and
then,
when
the
query
comes,
you
basically
find
out
where
those
fingerprints
are
and
Route
the
query
only
only
there.
So
then
you
can
have
say
C
advisor
metrics
that
are
local
to
the
cluster
that
sits
only
within
the
cluster
that
it's
actually
there
and
then
you
can
roll
up
stuff
and
then
store
it
at
the
region.
B
It's
wasteful,
it's
low
and
yeah,
so
from
from
a
technology
perspective
like
movie
from
the
old
model
into
the
new
model,
has
been
pretty
powerful
for
us
and
one
of
the
other
problems
that
we
had
was
that
how
would
you
do
routing?
So
we
have
a
very
reserved
label
like
a
tenant
ID.
We
we
call
it
namespace
every
series
that
comes
in.
We
look
at
the
namespace.
We
look
at
okay.
B
There
is
a
routing
table
where
this
namespace
goes
into
this
cluster,
which
means
that
when
you
query
as
well,
you
need
that
routing
key
to
say
that
okay,
this
namespace,
sits
on
this
cluster
I
find
out
into
all
shards
in
that
cluster.
This
becomes
problematic
from
a
native
Prometheus
experience,
because
you
can
no
longer
do
series
API,
you
no
longer
can
do
labels
API,
because
for
all
of
them
you
need
to
provide
namespace
and
semantically.
It
doesn't
work,
and
if
you
are
using
rafana,
then
you
basically
cannot
use
autocomplete.
B
A
lot
of
problems
existed,
so
the
pivot
would
be
again
to
go.
The
field
index
approach
have
a
arbitrary
query.
That
is
there.
It
doesn't
even
have
the
tenant
ID.
It
will
exactly
tell
you
where
to
look
for
the
data,
and
then
you
you
basically
so
we
became
more
Prometheus
compatible
by
by
pivoting
into
that
into
that
approach
as
well.
So
like
a
lot
of
a
lot
of
positives,
were
there
not
only
from
a
scale
perspective
but
from
a
user
usability
perspective
as
well,
which
basically
became
Planet
scale?
A
B
So,
like
I
mentioned,
we
have
the
concept
of
tenant
based
clustering,
so
right
now
what
we
do
is
like
we
have
like
a
default
cluster.
That's
there
where
pretty
much.
Everyone
goes
in.
That
gets
like
a
standard
retention
like
maybe
a
year
for
use
cases
that
have
higher
cardinality.
We
typically
try
to
tenant
them
on
different
clusters,
and,
given
that
a
cluster
is
its
own
physically
separated
entity,
we
can
provide
the
different
retention
for
that.
Some
might
have
one
one
some
might
have
six
months.
That's
that's!
A
B
B
They
see
the
data,
they
look
up
the
routing
table
to
figure
out
how
many
replicas
are
there
for
the
given
cluster.
So
if
it's
two
replicas
or
three
replicas
or
whatever
we
find
each
of
those
shards
to
write
into,
and
then
we
parallely
write
the
number
of
rights
that
basically
stick.
They
are
what
they
stick.
That
is
what
what
it
is.
Usually
all
three
rights
happen
without
any
issues,
but
when
we
query
out,
we
can
always
do
like
streaming,
merge,
dedup
and
then
make
sure
that
we
have
a
consistent
view.
B
At
least
when
we
we're
building
out
the
first
implementation,
cortex
was
still
actually
when
we
were
well
into
the
implementation
is
when
cortex
came
out
and
it
was
still
not
fully
production
grade
yet
and
by
the
time
they
became
more
mainstream
yeah.
We
were
like
well
into
the
journey
that
you
didn't
make
sense
for
us
to
converge
back
into
something
like
cortex.
Their
design
choices
are
interesting.
B
Some
of
the
some
of
our
design
choices
aligned
pretty
closely
to
them
as
well,
but
some
places
will
deviate,
for
example
like
they
have
the
ring,
and
if
one
of
the
shards
is
down,
they
keep
readjusting
the
ring.
In
our
case,
like
we
take
a
more
simplistic
approach
to
say
that
okay,
out
of
10
shots,
if
one
shot
is
down,
there
are
probably
still
two
more
replicas
that
are
there.
So
we
really
don't
care.
B
B
A
Yeah
that
that's
hurting
I
know
because
it
really
some
some
of
the
things
you
said
resonated
with
with
the
choices
that
cortex
made.
But,
as
you
said,
apparently
some
of
the
choices
aren't.
So
it's
valid
enough.
So
that's
that's
your
I
guess
metric
solution
that
the
one
that
you
mentioned
before
that
you're
considering
open
sourcing.
So
folks,
again,
if
you,
if
you're
interested
in
that,
do
a
follow
up
with
Vijay,
it
may
be
a
very
interesting
thing
to
collaborate
on.
A
B
B
We,
we
took
a
very
conservative
approach
in
the
sense
that
we
started
with
the
0.01
sample
rate,
so
one
in
ten
thousand
requests
gets
sampled
and
at
the
scale
at
which
a
company
like
eBay,
runs
like
it's
very
easy
to
get
a
sample,
even
if
you're
doing
like
one
in
ten
thousand.
So
we
started
getting
the
data
in
and
like
figuring
out
how
to
structure
our
pipeline
like
there
are.
B
There
are
God
knows
how
many
times
we
have
run
into
memory,
limited,
dropping,
droppings
fans
and,
like
my
teammates
Hoffman
and
Aishwarya,
like
they
have
been
added
to
figure
out
what
the
right
pipeline
should
look
like
and
I.
Think
like
we
finally
have
one
so
from
point
zero
one.
We
moved
to
point
one
one
percent,
and
now
we
are
getting
ready
to
move
into
two
percent
samples
and
the
idea
is
that
figure
out
a
sample
rate
that
is
good
enough.
100
sample
rate
is
probably
we
don't.
No.
B
My
opinion
is
that
no
company
should
should
need
the
100
samples,
but
we
are.
We
are
mindfully
moving
up
the
sampling
rate
to
a
point
where
it's
like
okay
Beyond
this.
We
really
don't
need
it
and
we
are
actively
working
with
our
SRE
Partners
to
figure
out.
Okay,
do
we
need
any
tooling
or
visualization?
Can
the
data
be
pulled
out
and
analyzed
in
such
a
way
that
it
supplements
the
analytics
that
we
have
on
top
of
our
metrics
platform,
our
events
platform
and
things
like
that?
B
So
it's
an
it's
an
it's
an
ongoing
Journey
for
sure
the
the
platform
engineering
route
helps
us,
like
you
mentioned,
to
standardized
attributes
in
a
way
that
everyone
gets
a
consistent
experience.
We
have
tried
our
hand
with
even
device
level
instrumentation,
so
there
is
a
project
on
our
eBay
open
source,
GitHub
repository,
which
basically
allows
you
to
do
iOS
instrumentation,
which
is
compliant
with
open
Telemetry.
B
So
we
have
device
level
spans
also
coming
in
so
it's
it's
easy
for
for
an
SRE
to
come
in
and
or
even
a
mobile
developer
to
come
in
and
say
that.
Okay,
this
is
my
service
for
for
the
device
and
every
interaction
that
actually
was
instrumented.
You
should
be
able
to
see
that,
along
with
all
the
actual
service
calls
that
are,
there
has
a
really
powerful
thing
like
which
we,
which
we
have
never
had
and
open.
C
B
I
think
we
have
some
knowledge
that
we
probably
have
that
we
can
share
so
looking
for
more
opportunities,
either
in
conferences
or
blog
posts
that
we
hopefully
will
will
have
in
some
of
the
learnings
that
we
have
had
in
tracing
is
a
is
it's
a
it's
an
interesting
and
pretty
challenging
Journey
that
we
have
been
through
I.
Think
that's
a
that's
an
accurate
way
to
summarize
how
things
have
gone
with
the
tracing
so
far,.
C
B
Our
journey,
while
it
started
with
tracing
tracing,
was
still
pretty
much
Alpha
with
the
early
customer
based
feedback
loop
that
was
in
the
serious
Journey
was
actually
with
metrics
because,
like
metrics
was
more
of
a
ga
platform,
it
directly
parse
time
to
detect,
which
means
that
any
hiccup
can
actually
blind
us
on
what
is
actually
happening
on
the
site.
So
from
that
perspective
like
we,
we
had
to
do
a
thorough
analysis
on
what
we
need.
What
what
would
scale
not
scale.
B
C
B
With
brahmendra
Singh,
he
runs
agents
here
in
San
Jose.
They
basically
worked
on
closing
some
of
the
feature
gaps
and
one
critical
feature
Gap
that
was
there
is
the
ability
to
do
a
dynamic
pipeline
reloading.
That
is
a
feature
that
we
used
to
rely
heavily
on
Beats,
because
whenever
a
pod
comes
up,
you
need
to
be
able
to
watch
the
APA
server
saying
it.
Okay,
this
part
has
come
up.
B
C
A
C
B
It's
a
stock
hotel,
along
with
plugins
that
that
that
that
that
we
add
in
and
then
ship
out
into
our
internal
clusters
and
one
of
the
receivers
that
we
put
in
is
this
file
file
reloader
receiver,
which
which
there
is
a
PR
of
stream
for
that
but
yeah
like
once.
That
feature
was
closed,
then,
basically,
it's
going
through
the
motions
of
thorough
testing,
so
premendra
Singh.
B
B
You're
gonna,
churn
cardinality
across
the
board
so
like
these
are
the
the
nuances
that
are
very
challenging
that
we
had
to
go
through
when
we,
when
we
did
this
like
there
is
in
fact
there
are
several
rollbacks
that
we
had
to
do
because,
like
okay,
we
found
this
new
bug.
Let's
go
fix
it
upstream
and
like
we
refuse
to
customize
the
receiver
internally.
It
would
have
been
very
easy
if
we
had
done
that,
but
then
that
has
like
long-term
repercussions,
so
the
agent
for
what
they
did.
B
B
So
once
those
were
taken
care,
then
then,
basically
it
was
like
go
clustered
by
cluster,
upgrade
everything
to
the
point
where
we
run
stable,
open,
Telemetry
collector
at
at
the
scale
at
which
we
do
right
now
and
like
we.
We
also
mindfully
do
like
quarterly
upgrades
to
make
sure
that
we
are
still
keeping
close
to
the
most
recent
version
and
that's
the
that's
a
very
interesting
part
about
open
television,
it's
so
crazy
in
the
amount
of
commits
and
contributors
that
that
it
has
it's.
B
A
Let's
talk
I
want
to
switch
over
from
from
eBay
to
the
community
and
one
of
the
most
fascinating
discussions
that
I
had
at
the
cube.
The
Europe
just
a
couple
of
months
ago
in
Amsterdam,
was
about
the
observability
query:
language
standardization
standard
for
query
line
querying
observability
across
logs
metrics
traces
and
potentially
other
signals.
For
me
that
was
a
very
exciting,
a
very
exciting
thing
to
see,
and
that
was
with
Chris
you
weren't
there.
A
Unfortunately,
you
weren't
able
to
attend
in
person
but
Chris
from
from
Netflix
with
your
your
partner
in
crime.
Let's
put
it
this
way,
so
he
was
there
and
we
had
a
very,
very
interesting
discussion
there
and
meeting
about
that.
So,
let's
start
with
the
basics,
tell
us
what's
it
about
and
what's
the
pain?
Why
do
we
even
need
that
and
let's
take
it
from
there.
C
B
So
the
the
the
query-
standardization
working
groups,
primary
Charter-
is
to
figure
out
the
landscape
of
observability,
primarily
from
query
standpoint.
So
we
have
an
established
standard
now,
which
is
project,
slash
vendor
neutral,
in
how
we
can
ingest
data.
But
on
the
query
side,
different
people
at
different
times
have
come
in
to
to
create
a
language
that,
at
certain
points
we
we
felt
made
the
best
sense.
But
there
are
too
many
options
right
now.
Everything
has
its
own
strength.
B
Compatibility
is
a
big
challenge
as
well.
So
the
idea
is
to
survey
those
language
creators
to
survey
end
users
to
see
what
are
the
pain
points.
What
are
the
things
that
worked
in
certain
languages
that
did
not?
What
did
not
and
things
like
that
and
to
come
up
with
a
database
of
all
this
information
which
could
potentially
be
used
to
come
up
with
a
recommendation
of
star
sorts
to
say
that?
B
Okay,
if
we
were
to
have
something
like
what
SQL
is
for
the
database
world,
this
is
probably
a
recommendation
that
we
can.
We
can
make
for
observability,
based
on
what
we
have
heard
from
customers,
saying
that
this
is
how
we
use
data
and
from
providers
to
say
that
these
are
the
technicalities
that
we
need
to
make
sure
that
when
you
actually
talk
to
storage
or
do
anything
on
behalf
of
the
customer,
this
this
makes
sense.
A
If
I
started
from
the
metrics
and
then
to
the
logs
and
the
query,
the
metrics
I
query
if
it's
Prometheus
with
prom
ql
or
maybe
influxql,
if
I
work
with
the
influx
ecosystem
and
so
on,
and
the
jumping
between
different
languages
and
different
dsls
and
different
semantic
ways
of
expressing
things.
This
is
a
pain
right
experience.
Why
did
you-
and
they
even
start
looking
at
this
this
page
so.
B
One
of
the
things
that
we
mindfully
decided
at
some
point
as
we
went
through
the
observed
the
journey
inside
of
eBay,
is
that
you
standardize
at
the
two
ends
so
only
on
ingest.
You
have
publicly
supported
open
contracts
that
are
there
and
on
the
query
side
also,
you
try
to
do
something
similar
to
the
point
where
it's
like.
B
If
a
new
piece
of
storage
technology
comes
in,
we
should
be
able
to
swap
the
technology
without
impacting
end
users,
because
for
large
organizations
migrating
from
one
standard
to
another
is
a
multi-year
journey
in
itself.
So
open
Telemetry
has
done
that
for
from
an
ingest
perspective,
so
even
even
within
the
realm
of
metrics,
while
Prometheus
is
just
is
more
or
less
the
standard.
That's
there
from
an
open
source
perspective.
There
are
so
many
open
source
clients
that
people
end
up
getting
fascinated
about.
B
Like
you
have
the
stock
Prometheus
client,
that's
there,
you
have
micrometer
client,
that's
there.
You
have,
you
probably
have
like
10
other
clients
that
people
people
are
really
happy
about
but
like,
if
you
have
like
thousands
of
microservices
that
are
running
within
your
company
like
having
everyone
use.
B
Their
own
sdks
is
a
bit
problematic,
because
if
you
want
to
duplicate
one,
it's
very
hard,
so
open
Telemetry
has
gotten
us
to
that
point
on
the
on
the
ingest
side,
and
we
want
to
do
something
similar
on
the
query
side
like
regardless
of
what
the
underlying
storage
is.
We
should
have
one
standard
language
in
which
we
can.
We
can
interface
with
the
system
and
that
problem
is
more
pronounced
on
the
query
side
than
on
the
ingerside.
B
If
the
project
for
some
reason
does
not
have
any
more
contributors,
then
basically
you're
stuck,
and
you
have
to
figure
out
how
to
migrate
into
a
different
one
and
migrate,
all
the
dashboards
that
are
there
as
well,
at
least
within
eBay,
like
we
have
I,
was
taking
stock.
A
couple
of
days
ago,
we
have
more
than
8
000
dashboards
that
are
there
like,
and
if
each
of
each
one
of
them
are
actually
using
an
interface,
that's
that's
defunct.
B
Now
it
becomes
a
big
problem,
so
that
is
one
of
the
things
like
have
the
a
consistent
experience
between
logs
metrics
and
traces
like
interoper
operability
is
still
a
problem.
That's
10
steps
ahead,
but
have
a
consistent
language
and
the
other
thing
is
like
every
system
or
a
medium
to
large
organization
can
always
have
multiple
observability
providers.
B
It's
a
very
natural
thing,
like
you
might
be
using
a
project
for
its
strings
for
a
portion
of
your
use
cases
and
you
might
be
using
a
product
for
its
strings
on
a
limited
scale
in
some
other
use
case.
But
how
do
we
interoperate
across
them?
If
you
take
metrics
like?
Should
the
end
time
end
time
be
included
or
excluded,
like
one
project
might
decide
something
and
the
product
might
decide
something
other
than
that.
B
Interface
like
there
are
so
many
places
where
a
standardized
query
language
makes
a
lot
of
sense
not
only
from
an
interoperability
perspective
but
from
a
standard
for
everyone
to
adopt
so
I
think
like
that
is
a
problem
that
a
few
of
us
pitched
in
kubecon
North
America
during
one
of
the
open
television
discussions,
have
been
pretty
gracious
in
facilitating
the
conversation
to
the
point
where
we
have
a
working
group.
So
anyone
who
finds
this
as
a
problem
like
I
strongly
urge
you
and
welcome
you
to
come,
join
the
working
group.
A
Yeah
and
just
to
for
the
audience
is
not
familiar
with
with
with
the
members
that
you
mentioned.
Essentially,
the
working
group
started
under
tag
observability,
the
technical
Advisory
Group
for
observability
of
the
cncf.
So
now
it's
an
independent,
it's
a
it's
a
its
own
working
group
after
the
TOC
approved
the
charter
and
and
it's
fairly
new,
so
I
think
that
the
first
meeting,
a
call
for
the
working
group
was
the
end
of
May
right,
the
30th
or
something
like
that
of
May.
C
B
I,
don't
remember
the
slack
channel
name
on
top
of
my
mind,
but
we
have
a
slack
channel
in
in
cncf
slack.
You
can
probably
put
it
in
the
comment
section
after
this
and
we
meet
first
Thursday
and
so
sorry,
first
Tuesday
and
third
Tuesday
9am
PST
and
second
and
fourth,
nine
PM
PST.
So
that's
our
Cadence
so
that
we
have
both
an
America's
friendly
and
an
Asia
Pacific
friendly,
Time
Zone
on
time
in
which
we
can
meet
and
collaborate.
B
A
You
have
it
here,
I'll
also
paste
it
as
part
of
the
show
notes
for
the
listeners
on
the
podcast
and
yeah
I,
really
really
encourage
you
to
first
check
out
the
slack
Channel
and
check
out
the
charter,
and
there
is
a
running
Google
doc
like
any
other
working
group
with
the
nodes
you
get
now
the
first
meeting
and
more
to
be
added
so
chime
in
join
the
zoom
and
on
the
time
zone
that
is
friendly
for
you
really
exciting
and
I.
Don't
know.
I
discussed
this
Vision.
A
Also
I
I
talked
about
unified
I
talk
a
lot
about
unified,
observability
and
I,
emphasize
that
unified
observation
doesn't
end
just
with
the
instrumentation
and
collection
side.
It's
it
goes
to
the
back
end
and
I
think
you
took
the
First
Step
Beyond,
just
instrumentation
and
and
the
generation
and
the
collection
side
of
the
house.
A
C
B
B
A
I'm,
counting
on
on
catching
up
with
you,
there
I,
missed
you
on
the
on
the
at
kubecon
Euro,
but
definitely
looking
forward
to
a
cubecon,
North,
America
and
Chicago
and
and
your
talks,
it's
great
to
see
you
taking
the
stage
more
and
more
with
scale
20x
and
then
others
you
have
so
much
to
share.
We
just
scratch
the
surface.
With
this
episode,
but
you're
doing
at
eBay,
your
team
is
doing
amazing
stuff,
so
I
look
forward
to
hearing
more
and
your
blogs
follow
for
the
listeners.
A
Do
follow
the
the
blog
post,
fascinating
about
different
aspects,
both
on
the
metric
side
on
the
hotel
side
and
many
others,
so
lots
of
good
stuff
coming
out
of
this
team
keep
an
eye
out
for
this.
One
I'll
share
a
few
links
also
on
the
show
notes
for
some
interesting
topics
that
we
discussed
here
and
I
want
to
with
that.
Thank
you
very
much
for
joining
me
on
this
show
Vijay.
Thank
you
very
much.
B
So
much
one
thing
before
we
close
like
eBay
is
always
looking
to
hire
the
best
individual
Engineers
out
there.
Please
do
check
out
our
jobs
page
if
you
are,
if
you
were
intrigued
by
the
scale
at
which
we
work
at
and
the
problems
that
we
solve
like
I,
consider
it
to
be
an
absolute
god-given
blessing
to
have
spent
my
entire
formative
years.
I've
been
this.
This
month
marks
11
years
at
eBay.
My
first
job
eBay
has
been
great
I
so
like
if
you're
looking
for
something
challenging
eBay
is
the
place
to
be.
A
A
For
today's
episode
of
open
observability
talks,
all
the
episodes,
as
always
are,
will
be
available
on
the
favorite,
your
favorite
podcast
app
or
on
YouTube
We
publish
the
episodes
on
the
last
Thursday
of
the
month,
so
this
episode
will
also
come
out
to
the
podcast
apps
last
Thursday
of
the
month.
If
you
are
listening
to
this
episode
on
on
the
podcast,
do
know
that
we
live
stream,
the
episodes
on
Twitch
and
YouTube.