►
From YouTube: Loki: An OpenSource Zipkin / Prometheus Mashup, Written in Go [I] - Tom Wilkie, Weaveworks
Description
Loki: An OpenSource Zipkin / Prometheus Mashup, Written in Go [I] - Tom Wilkie, Weaveworks
In this talk I'll present a prototype OpenTracing implementation dubbed "Loki". Loki is a Zipkin-compatible distributed tracer written in Go. Unlike Zipkin, Loki pulls traces from the application under examination by reusing Prometheus' service discovery and scraping code. This presents some unique advantages over traditional distributed tracers as the tracer knows the identify of the services under examination, and helps present a consistent naming topology when used with Prometheus.
About Tom Wilkie
Tom is a Software Engineer at Weaveworks. Previously he was at Google as a Site Reliability Manager for Google Analytics. Before that he was Founder, VP Eng and CTO at Acunu, and before that a Software Engineer at XenSource. In his spare time, Tom likes to make craft beer and build home automation systems - and distributed tracers!
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Okay,
I
think
we'll
get
started.
I
think
we
should
start
things
promptly:
I'm,
probably
going
to
eat
all
the
time.
So
hello,
everyone,
my
name
is
Tom
I'm,
a
software
engineer
at
a
company
called
weave
works.
But
actually
this
is
a
personal
project
of
mine.
The
relationship
to
the
work
is
relatively
minimal.
My
day,
job
is
actually
working
on
a
distributed
version
of
prometheus
and
so
I
guess
it's
kinda
related,
but
yeah
I
have
a
talk
entitled
Loki
and
Zipkin.
A
Sorry
Loki
a
Zipkin,
Prometheus
mashup,
if
I
actually
say
mash
up
with,
without
like
with
a
straight
face.
No
because
it's
a
bit
silly
but
whatever,
let's
start
with
a
few
questions
for
everyone,
so
hands
up.
Well,
hopefully,
from
the
last
talk
you
will
know
what
open
tracing
is
so
hands
up
everyone
and,
if
you
don't
know
open
tracing
is
like
what
are
you
doing
here?
A
This
is
not
an
introduction.
We
talk
so
hands
up.
You'll
know,
open
tracing
is
I'm
going
to
so
how
many
people
have
heard
about
for
me.
Just
keep
your
hands
up
now.
Put
your
hands
down.
If
you
don't
talk
to
me,
kisses
I'll
tell
you
all
about
third,
not
bad
again.
Who's
heard
of
weave
works
and
Oh
more
people
than
Prometheus
us
interesting,
not
normally
like
that
ok
cool.
Well,
we
must
be
doing
a
good
job,
I'll
start
by
so
I've
written
my
own
tracer.
Why
would
I
do
that
by?
What
is
wrong
with
me?
A
Well
cortex.
Is
this
distributed
prometheus
that
I
work
on
in
my
day,
job
and
I
was
having
some
performance
issues
with
it
and
the
job
I
bawled
it
out
a
little
bit
more
complicating
this,
but
there's
a
job
called.
We
called
the
distributor
and
it
writes
to
the
injustice,
and
it
writes
in
parallel
to
multiple
investors
to
achieve
replication,
so
the
distribution
of
samples,
it
writes
them
in
parallel
to
at
least
three
investors,
and
then
it
returns
and
it
returns
when
at
least
a
quorum
of
those
and
jester's
has
successfully.
A
You
know
acknowledged
the
right
in
this
is
if
anyone
familiar
with
like
dynamo
style
consistency,
it's
basically
that,
but
we
had
this
problem.
This
is
a
Prometheus
crop.
We
have
this
problem
up
here
where
the
distributors
were
reporting
a
99%,
our
latency
of
about
80
milliseconds
and
the
in
jesters
were
for
the
same
operation.
Reporting
8ye,
given
the
scheme
I've
just
described,
I
was
kind
of
expecting
the
distributor's
to
have
the
same
or
even
a
potentially
better
latency
profile
than
the
injustice,
because
a
slow
in
gesture
won't
slow
down
a
batch
right.
A
A
The
reality
is
I've
been
trying
to
build
a
distributed
tracer
for
a
few
years
now,
I
really
like
the
technology.
My
first
attempt
was
this
thing
called
weave.
Tracer,
it's
got
a
nice
UI,
the
chappie
did
the
UI.
He
also
did
all
our
product.
You
eyes
is
talking
next
door
into
talks,
time
highly
recommended,
but
it's
at
the
prototype,
weave,
tracer,
I
wrote
on
holiday
in
Singapore
and
it
the
concept
behind.
This
was
it
didn't,
require
application
instrumentation.
So,
unlike
all
these
open
tracing
stuff
you've
been
hearing
about,
this
was
supposed
to
be
magic.
A
It
used
P
trace,
which
is
a
Linux
system
thing.
Linux
kernel
thing
to
intercept
system
calls
between
applications
of
the
kernel
and
then
it
kind
of
infer
application
behavior,
and
we
had
a
demo
that
worked
for
some
very
very
tightly.
You
know
defined
version
of
work,
I
mean
the
reality.
Is
it's
not
a
general
technique,
so
we
kind
of
put
it
to
one
side
and
I
went
actor.
You
know
my
normal
day
job
and
why
the
name
low-key,
for
my
own
trace
of
well
as
you'll,
see
it's
quite
heavily.
Inspired
by
Prometheus
inand.
A
This
chap
on
the
internet,
who
must
be
right,
thinks
that
Loki
is
the
Norse
equivalent
of
Prometheus
I've
only
read
the
first
few
lines
of
the
paper,
so
I
have
no
idea
if
he's
right
or
not,
but
I
like
the
name,
Loki
I'm,
also
a
big
Marvel
fan.
So
have
you
any
questions
feel
free
to
just
interrupt
what
makes
Loki
different
to
your
standard,
distributed
tracer
and
I'm
going
to
take
Zipkin,
because
the
only
one
I
know
and
and
if
I
happen,
to
have
happened
upon
something
that
all
the
other
distributed.
A
Chasers
do
that
I
didn't
know
about.
I
am
NOT
going
to
apologize,
so
Prometheus
is
to
graphite
another
common
monitoring
system
as
Loki
is
to
Dickon.
So
this
is
a
common
like
11-plus
style
statement.
So
what
makes
Loki
different
addiction
is
the
same
concept
that
makes
Prometheus
differences
if
attila
graphite-
and
this
is
just
the
concepts
of
pushing
versus
pulling
relatively
straightforward
in
in
the
Zipkin
world-
your
act
push
spam
two-zip,
they
send
them,
they
have
to
know
it
lives
and
in
the
Prometheus
world,
Prometheus
goes
out
and
pulls
metrics
from
your
jobs.
A
So
Prometheus
knows
where
your
jobs
are,
it
goes
in
collecting
them
and
it
connects
them
and
collects
metrics
I.
Don't
really
want
to
get
into
a
big
push
versus
pull
debate.
I'm
a
fan
of
the
pull
model,
but
I
think
there
are
valid
arguments
on
both
sides.
There
are
two
fantastic
like
facts
and
blog
posts
that
go
into
this
in
more
detail,
and
if
you
want
to
argue
about
it,
then
we're
going
to
get
a
beer
and
argue
about
it.
A
Let's
not
do
it
here
we're
going
to
a
bit
more
detail,
then
so
graphite
you've
got
your
jobs
there
instrumented
to
collect
metrics.
They
know
we're
great
we're
graphite
lives.
I
should
graphite
the
right
thing,
order
actually
used
it
and
they
push
metrics
to
graphite.
So
in
this
model,
so
it's
claimed
relatively
easy
to
overwhelm
a
graphite
server
with
too
many
metrics,
and
so
you
end
up
a
little
aggregated
in
the
middle
and
so
on.
A
Plus
all
your
jobs
need
to
know
where
graphite
is
and
if
you
want
to
run
more
than
one
graphite,
say
your
graphite
developer
and
you
want
to
have
a
your
own
little
graphite
running
on
your
laptop.
Then
you
get
your
jobs
to
push
the
both
or
I'm,
not
sure
how
that
works,
but
in
the
prometheus
world,
Prometheus
knows
where
your
jobs
are
and
we'll
get
into
why
it
knows
and
how
it
knows,
and
it
goes
out
to
your
jobs.
It
connects
to
them
it's
great
to
the
metrics,
and
then
it
does
its
thing.
A
This
allows
Prometheus
to
a
kind
of
back
off
when
it's
getting
overwhelmed.
You
know
if,
if
it's
got
struggling
to
write
discs
or
something,
then
it
just
won't
scrape
metrics,
so
it
kind
of
naturally
has
this
feedback
loop,
but
also
this
allows
Promethean
allows
you
to
run
a
Prometheus
on
your
laptop,
so
I
do
this
all
the
time
because
I
develop
on
Prometheus.
So
if
I
want
to
run
a
Prometheus
and
if
I
want
to
write
some
code,
I,
don't
just
push
the
code
to
my
production
server
and
break
our
monitoring.
I
run
out.
A
My
laptop
and
I
broke
my
laptop
instead
and
my
nap
toxin
go
and
scrape
our
production
jobs,
and
so
the
the
key
thing
here
with
the
Prometheus
model
is
Prometheus
needs
to
know
where
all
your
jobs
are
and
what
the
identity
of
them
is.
So
you
can
do
this
with
a
massive
config
file,
but
no
one
does
like
what
you
really
want
to
do.
A
Is
you
use
Prometheus's
service
discovery
integrations,
and
this
is
where
from
ETS
will
go
out
and
talk
to
your
infrastructure,
discover
where
the
jobs
are
dynamically
and
build
its
own
little
table
of
targets,
and
then
you
know
do
its
thing,
and
this
is
great.
This
has
integrations
for
cuba
Nettie's,
so
that
if
you're,
changing
your
cuba,
nephews,
if
you're
scaling
jobs
up
and
down
prometheus,
just
watches
the
the
correct
pod
objects
or
whatever
objects
in
Suva
Nettie's
sees
that
there's
new
ones
goes
into
nexus
and
start
scraping
them.
You
never
have
to
change
a
line.
A
A
We
run
a
lot
of
memcache
--is
for
different
things
and
I
don't
want
to
have
to
tell
memcache
a
over
here
that
this
is
called
the
scope,
memcache
and
then
cache
be
over
here
that
it's
real
name
is
called
the
cortex
memcache,
because
Prometheus
already
knows
that
this
one's
the
scope
one
and
this
one's
the
cortex
one.
So
we
just
used
exactly
the
same
config
in
different
contexts
and
Prometheus
can
pick
that
up.
A
So
we
have
Zipkin
versus
Loki
the
push
versus
the
pull
model
that
we
have
graphite
versus
delicious,
and
so
you
know,
as
I'm
kind
of
building
up
to
the
Zipkin
versus
Loki
Zipkin
I
think
is
a
push
based,
modeler
and
Loki.
Is
this
pull
based
model,
and
this
has
some
certain
benefits
kind
of
go
into
in
the
demo?
Well,
I've
kind
of
gone
into,
but
I
think
it's
kind
of
cool.
So
what
does
this
look
like?
A
Well,
I
wrote
a
very,
very
small
library
that
you
have
to
embed
into
your
application,
and
so
in
the
previous
talk
they
talked
about
how
you
have
to
eventually
choose
the
trace.
You
want
to
use
all
of
the
integrations
in
your
application
or
open
source
and
standard,
but
eventually
you
say
right,
I'm
now
just
going
to
instantiate
my
trace
it,
and
if
you
look
at
our
code,
this
is
like
one
line
where
it
says:
Loki's
new
tracer
register
global,
something
and
inside
this
tracing
library,
I've
written,
is
just
a
circular
buffer.
A
That's
capped
to
a
certain
size.
You
know
a
thousand
traces,
a
thousand
traces
I
think
and
a
HTTP
handler
that
allows
Loki
to
come
along
and
pull
those
chases
out
of
your
job
and
then
for
actually
all
of
the
kind
of
complicated
open
tracing
bits.
I
just
literally
used
all
of
the
zip
code,
because
you
know
this
is
you
know,
maybe
a
hundred
lines
of
code,
so
pians
got
loads
of
logic
on
how
to
deal
with
traces
and
so
on.
A
The
quiet
library
keeps
pending
spans
in
memories
in
a
ring
buffer,
so
if
it
gets
full,
it
starts
dropping
them
off
the
end
of
the
ring
buffer.
The
traces
HTTP
handler
grabs.
All
these
in
memory,
traces
and
spam-
sorry
serializes
them
using
swift,
because
I'm
just
using
the
zip
in
code
and
yes
spans,
will
be
dropped.
Then,
if
I
focus
in
on
the
low-key
piece,
oh
there's
an
animation.
Let
me
know
that.
Okay,
if
I
focus
in
on
the
Zipkin
piece
on
the
low-key
side,
we've
lifted
the
Prometheus
retrieval
library.
A
This
I've
opened
a
PR
to
just
add
a
little
callback
in
so
I
can
implement
my
own
script
function
and
then
you
know:
I
provided
a
script
function
that
understands
the
thrift
Zipkin
format
that
we're
sending
this
again
like
the
Prometheus
life
is
huge,
does
loads
of
things
it
talks
to
Azure.
It
talks
to
me
sauce.
It
talks
to
console
it
talks
to
Cuban
nettie.
That
talks
to
you
know
is
a
huge
list
on
their
website.
The
integration
so
they've
spent
a
lot
of
time
and
and
a
really
high-quality
now
and.
A
Yeah,
this
is
a
great
library.
My
integration
with
this
library
is
another
hundred
lines
of
code.
It
didn't
take
long,
then
there's
some
in
memory
storage.
This
is
kind
of
the
weakest
bit
low
key
just
does
everything
in
memory
it
stores
I've
got
the
constants
set
to
1
million
right
now,
so
it
stores
a
million
traces
and
then
there's
a
doe
reimplementation
of
a
Zipkin
api
against
this
storage,
which
is
only
a
few
hundred
bunch
code
and
then
because
I'm,
not
a
UI
guy
and
our
UI
guy
doesn't
know
about
this.
A
A
So
this
is
kind
of
cool,
because
there's
this
concept
in
so
I'm
going
to
pop
back
out
now
the
Prometheus
library
has
this
concept
called
relabeling
brian
is
very
keen
on,
and
this
is
where
you
can
basically
use
reg,
X's
and
various
patterns
to
construct
meta
information
about
the
targets
that
gets
gathered
up
into
this
big
dictionary,
and
then
certain
keys
have
special
meanings
and
you
can
basically
use
these
special
meanings.
So
a
good
example
would
be
the
job
and
the
instance
key
within
these.
A
Within
these
dictionaries,
so
I
have
developed
over
you
know
the
couple
years
we've
been
running
our
cluster
certain
standards
and
practices
for
how
we
like
to
name
our
jobs.
You
know
this
is
an
opinionated
thing.
This
isn't
something
I'd
expect
you
guys
to
to
want
to
do
exactly
how
I
want
to
do
that.
This
is
how
I
want
to
do
it
and
I
want
the
names
of
my
jobs
and
my
services
to
be
the
same
in
my
prometheus
cluster
as
they
are
in
my
Loki
thing.
A
This
also
means,
as
I
touched
on
earlier,
the
jobs
don't
need
to
know
what
their
name
is,
so
it
means
I
can
reuse.
You
know,
we've
got
a
standard
template
for
making
a
memcache.
We've
got
a
standard
template
for
making
a
console
cluster
standard
template
for
an
app
cluster
that
we
just
we
just
bash
bash
cookie.
Cookie
cutter,
you
know
cut
out
the
in
memory.
Storage
implementation
is
naive,
is
my
polite
way
of
describing
it.
Queries
are
really
slow
right
now,
because
it's
just
a
for
loop
that
goes
over.
A
Everything
can
sees
what
matches
that's
so
much
low-hanging
fruit
there
that
you
know
I'll
get
around
to
at
some
point.
Yeah
typical
API
is
the
API.
Is
it
can
compatible,
and
that
makes
means
we
can
use
the
UI
and
it's
all
open
source.
This
is
a
personal
project,
so
we
put
it
in
the
Weaver
works,
experiments,
folder
and
it's
all
written
and
go
where
to
say
cute
little
mascot,
because
we
were
mainly
a
go
company.
A
It's
about
all
in
I,
know,
I,
said:
there's
100
lines
here
and
a
hundred
lines
they're
all
in
it's
about
one
and
a
half
thousand
lines
of
go.
So
it's
not
a
lot.
In
my
opinion,
this
sounds
great,
whereas
the
catch
so
as
I've
kind
of
alluded
to
this
is
a
prototype.
We
do
use
it
in
our
development,
environment
extensively
and
I.
A
Well,
currently,
my
client
library
doesn't
work
like
that
just
yet,
but
that's
like
just
a
bug:
the
crew
performance
sucks
again,
that's
just
a
really
naive
for
loops
like
just
to
continue
in
there
or
breaking.
They
would
probably
fix
that,
and
the
single
up
process
architecture
of
Loki
is
probably
going
to
be
a
limiting
scalability
but
I.
You
know
I.
A
Think
prometheus
is
a
great
example
of
what
you
can
achieve
with
a
single
process
like
most
people
a
single
process,
Prometheus
server
will
be
more
than
enough
for
them,
no
matter
how
big
dome
structure
for
for
Loki
I
think
we
can
probably
do
something
similar,
but
you
know
we're
not
pushing
it
yet.
Our
dev
cluster
is
pretty
small
and
then
there's
just
kind
of
more
interesting
topics
that
previous
speaker
touched
on,
that
sometimes
spans
won't
get
delivered
and
that
this
leads
to
incomplete
traces.
A
Well,
interestingly,
in
the
push
versus
pull
debate,
I
think
pull
probably
makes
that
worse,
and
this
is
because
you
know
if
you've
got
a
really
high
frequency
service.
You
know
there's
going
to
be
filling
up
and
dropping
spans
out
of
its
ring
buffer.
When
you
start
your
Loki
server,
all
of
the
scrapes
will
be
aligned
more
or
less
and
will
scrape
everything.
A
Similarly,
and
because,
because
they're
close
in
time,
they'll
probably
associated
you
get
nice
complete
traces
but
as
it
goes
on,
the
scrapes
will
get
a
bit
of
jitter
and
will
start
to
span
out
and
start
to
not
overlap
and
you'll
pull
one
set
of
spam
from
this
service.
Another
set
spans
in
this
service
you're
going
to
the
joint
and
nothing
will
join
so
I,
think
there's
a
problem.
There
I
really
had
a
chance
to
dig
into
it,
I
just
fixed
it
by
making
the
scrape
interval
less
and
less
until
it
went
away
so
yeah.
A
So
back
to
that
cortex
performance
issue.
We
we
have
this
screen.
We
have
the
distributor
writing
to
the
in
jesters
and
we
saw
distribute
to
high
right
latency
and
finally,
two
weeks
after
I
set
out
solve
this
problem.
I
finally
had
the
distributor,
tracer
working
and
I
got
a
trace
out
getting
my
first
trace
out.
This
was
was
quite
a
rewarding
experience
and
we
saw
is
the
distributor.
So
it's
doing
you
know,
we've
got
some.
A
We
call
it
the
author,
it's
our
front,
end
service
that
does
authentication,
hence
the
name,
and
you
can
see
we're
doing
the
handle
request,
which
is
basically
the
thing
in
the
distributor
that
receives
the
samples,
there's
a
big
gap,
and
then
we
send
the
samples
to
the
injectors
and
the
injectors
kind
of
respond
pretty
quickly
and
we're
done
so.
I
really
wanted
to
know
what
was
in
the
big
gap
and
so
I
wouldn't
bill.
A
I
went
and
put
a
couple
of
spans
in
there,
so
we've
got
a
little
internal
library
inside
we've
works,
which
you
can
wrap
any
piece
of
code
with
this
library
and
it
will
produce
some
spans
for
you
and
it
will
produce
some
metrics
for
you
and
it'll
do
some
logging
for
you,
and
this
is
basically
exactly
what
the
previous
guy
was
talking
about.
It's
really
really
powerful,
so
we
just
went
and
did
this
deployed
new
version
and
we
saw
that
all
the
time
was
spent
in
unmarshal
in
the
protobuf
request,
and
I'm
like
that.
A
Not
right,
protobufs
are
supposed
to
be
fast
and
it
turns
out.
The
governor
implementation
of
proto
bus
is
reflection,
based
which
surprised
me
a
little
bit
because
I
thought
the
whole
point.
Why
am
I
generating
all
these
data
structures
and
doing
all
this
code
generation
when
we're
still
relying
on
reflection
anyway?
We
found
I
found
this.
We
put
this
project
called
Go,
Go
Pro,
so
I
don't
know
if
anyone's
heard
of
it
absolutely
fantastic
I
switched
to
using
that
I
enabled
all
of
its
optimizations.
A
You
know
without
thinking
and
this
problem
just
went
away,
so
I
did
actually
spend
a
little
bit
more
time
like.
Oh,
we
also
did
a
an
optimization.
The
root
of
this
problem
was
that
the
the
values
you're
sending
to
the
distributor
and
I
don't
know
if
you
know
Prometheus,
but
it's
lots
of
label
value
pairs
and
those
strings
tend
to
be
reasonably
small
and
lots
and
lots
and
lots
of
small
strings
doesn't
really
fit
very
well
with
a
go.
A
Garbage
collector
I
know
go
talks
about
how
good
its
garbage
collectors
but
I
spent
too
much
of
my
life
fighting
it.
So
it
turns
out
it's
relatively
straightforward,
to
not
allocate
strings
for
that
and
just
use
the
underlying
byte
buffer.
If
you,
you
know,
use
the
unsafe
package
and
that
literally
got
rid
of
this
problem,
and
so
this
is,
this
is
with
the
fix,
and
now
it's
like
by
the
end
of
this
day
is
getting
quite
late.
A
I
couldn't
believe
that
the
monitoring
was
telling
me
it
was
25
milliseconds
because
it
was
like
80
to
100
before
so.
I
reverted
back
to
the
master
branch
and
I'm
like
yeah
yep,
it's
100
and
I
bend.
My
deployed
the
fixin's
into
death
and
I
was
just
so
happy.
So,
let's
see
if
we
can
do
a
demo
just
bear
with
me,
while
I
reconfigure,
my
screen.
A
A
Right
then,
so
if
anyone
I've
pre
cached
the
pages,
because
the
Wi-Fi
has
been
so
flaky
but
I
will
try
and
do
a
live
one.
So
this
is
the
zipped
in
UI.
I
didn't
write
this,
but
it's
talking
to
Loki,
which
I
did
right
and
as
you'll
see
here.
These
are
the
list
of
jobs
that
we've
got.
Let's
make
that
a
bit
bigger
for
you.
These
are
the
list
of
jobs
that
we've
got
instrumented
with
Loki
you'll
notice,
a
particular
pattern
here:
I
do
namespace
slash
job
name
or
in
this
case.
A
In
this
case,
this
is
like
a
service.
Basically
now
the
reason
I
did
that
was
previously.
We
only
have
one
namespace
everything
within
the
default
namespace
and
we
were
running
a
memcache
and
then
we're
like.
Oh,
we
need
another
memcache
and
I
still
want
to
kind
of
called
it.
Memcache,
so
I
create
a
new
note,
a
newer
new
namespace
and
called
it
memcache.
But
then
all
of
my
old
rules
and
all
my
old
dashboards
or
aggregating
all
the
stats
for
both
memcache
is
so
I'm
like
yeah,
okay,
we're
not
going
to
do
that
anymore.
A
You
please
read
our
new
live
demos,
but
yeah.
Here's,
basically
the
distributed,
trace,
I.
Think
the
clocks
out
on
this
machine,
because
this
is
supposed
to
be
up
here
and
I,
haven't
quite
figured
out.
Why
they're
overlapping
with
this
thing,
not
quite
pretty
as
I,
was
hoping.
Let's
see
if
a
different
one
is
better.
If
anyone
knows
how
the
UI
is,
it
can
works
and
can
tell
me
what
this
bug
is
like.
Oh,
you
know
why
you
were
there
yeah
same
again,
but
it
is
basically
a
distributed
trace.
A
This
is
what
a
real
life
this
tree
of
trace
looks
like
because
it's
never
as
clean
as
they
look
in
the
unlikely.
You
know
their
normal
presentations,
but
you've
got
the
HTTP
GET
coming
into
the
query
err.
Then
it's
doing
some
query
pages
to
DynamoDB
and
it's
doing
some
Oh.
Actually,
as
the
previous
example
was
for
right,
this
is
for
query.
Then
it's
doing
a
query:
don't
distribute
the
night
school
distribute,
so
it's
not
the
distributor.
A
It's
the
query
service,
doing
a
query
to
the
query
service,
which
also
goes
in
parallel
talks
to
the
adjusters
and
then,
when
the
query
pages
return
to
go
and
talk
to
memcache,
and
then
we
return
to
the
user.
So
this,
like
I've,
been
I've,
been
optimizing.
The
queries
and
that
query
that
we're
looking
at
there
took
okay
took
14
milliseconds,
which
is
not
bad
I.
Quite
like
that.
A
So
yeah,
that's
that's
it
can
one
of
the
things
I
did
want
to
show
you
guys
this
isn't
a
product
plug
plug
I
promise,
but
if
we
go
and
look
at
our
dev
environment,
I
just
want
to
show
you
that
the
identity
and
our
monitoring
system
is
the
same
as
the
identity
in
our
tracing
system.
So
we
go
to
our
monitoring
system
and
what
were
we
looking
at?
A
Don't
we
so
we
want
to
know
the
cortex
memcache,
actually,
let's
just
type
memcache,
yet
there's
all
the
messengers
cortex
memcache
duration
seconds
bucket,
and
then
we
know
that
the
job
was
querier
and
there
you
go.
You
can
see
it's
sort
of
it's
more.
You
can
see
that
the
identity
for
the
job
is
the
same
as
the
identity
for
the
traces
and
just
to
complete.
This
I
will
do
some
rate.
A
So
that's
this
UI
is
the
thing
that
the
other
chaps
talking
about
two
to
two
stalks
time,
but
that
is
the
99th
percentile
latency
for
memcache
memcache
gets
in
the
last
hour
and
the
identity
of
the
job
and
the
identity
of
the
operation
is
the
same
between
tracing
and
between
the
demo
works
cool.
Okay,
that's
the
demo
and
what
am
I
going
to
say
next,
so
there's
big
to-do
list.
A
If
anyone
would
like
to
help
me
with
this,
please
come
and
help
me
I
want
to
make
it
support
multiple
scraper,
so
I
can
actually
do
the
gentleman
on
my
laptop
I
want
to
move
away
from
thrift.
Two
protos-
and
this
is
just
a
personal
taste
thing-
I,
believe
thrift
predates
proto's
and
it's
open
sourcing,
so
I
think
that's
wise.
It
comes
on
twist,
but
I'm.
You
know,
I've
got
experience
with
protests
now
I
want
to.
A
A
If
you
go
to
it
in
a
web
browser
to
not
give
you
a
binary
thrift
representation,
but
to
give
you
like
a
nice
useful,
HTML
page,
so
that
this
client
library
becomes
useful
on
its
own,
like
if
you've
got
a
single,
a
single
binary
application
that
doesn't
do
distributed
tracing,
but
you
still
want
to
go
and
see
traces
on
demand.
Then
you
could
use
this
as
I
said
everything's
in
memory
right
now,
so
I
want
to
do
local
storage
for
bolt
they're,
probably
using
both
dB
I
want
to
make
queries
faster.
A
Hopefully
that
will
solve
like
them
and
maybe
make
you
know,
do
a
distributed
cloud
storage
version.
Maybe
not
so
we
also
write
out
there.
What
was
supposed
to
say
here?
Oh
yes,
coming
back
to
the
question
of,
why
did
I
write
my
own
tracer
and
I
know
I've
had
like
a
thing
for
tracing
and
I
really
wanted
to
do
this
for
a
couple
of
years
now,
but
the
reality
is
why
I've
done
it
now
and
why
I've
been
able
to
do
it?
Now?
A
Is
because
I
can
because
of
open
tracing
like
I,
didn't
have
to
write
client
libraries
I
didn't
have
to
write
a
transport
form,
I
could
just
reuse.
This
I
didn't
have
to
go
an
instrument.
My
G
RPC
libraries
I
didn't
have
to
instrument
the
go.
Http
lie
because
that
all
exists
with
open
tracing
I
mean
building
this
standard,
as
API
has
really
enabled
these.
You
know
little
weekend
hockey
projects
which
makes
me
happy.
So
thanks
guys.
Whoever
did
that
like
really
good
idea.
Thank
you.
A
A
My
words-
and
this
is
the
project
that
I
work
on
and
I've
been
instrumenting,
and
if
we
go
and
look
in
that
piece
of
the
storage
code
chunk
store,
then
we
will
see.
Oh
no,
it's
not
in
there
we've
recently
refactored
AWS
Storage
client.
So
we
basically
for
things
like
AWS
we've
instrumented
the
clients,
because
they'd
abuse
has
an
instrumented
I,
think
that's
his
ex
tracing,
which
maybe
one
day
we'll
be
able
to
use.
But
and
then,
if
we
scroll
down
this
is
the
Prometheus
metrics
we've
declared
and
then
boo-boo
this
not
it
yet
simple.
A
One
would
be
where
we
put
a
chunk
here.
We
go
so
that's
the
code
where
we
put
a
chunk
into
s3.
You
don't
really
need
to
know
what
Junkers
and
that,
basically
the
instrument
code,
the
instrument
that
time
request
histogram
function
takes
a
couple
of
parameters
like
the
name
of
the
span
and
the
context
and
the
Prometheus
variable
and
then
returns.
Your
closure
takes
a
closure,
sorry
or
function.
A
A
You
free
right,
yeah,
that's
a
good
point.
I
didn't
go
in
to
thank
you.
The
question
was
from
a
given
application.
I'm
requesting
this
trace
at
endpoint
is
requesting.
Traces
was
the
question
and
is
that
choice,
partial
or
complete,
and
so
the
actual
thing
isgetting
expands
is
the
individual
spans
and
the
joining
up
of
the
trace
happens
in
the
loci
server.
Realistically,
like
that's
like
one
line
of
code
like
you
just
it's
just
a
sort
right.
A
The
every
span
has
a
common
trace,
ID
and
a
distinct
span
ID
and
then
in
the
loci
server
I
just
group
together,
things
that
have
the
right
span,
ID
and
I
sort
them
by
time,
and
then
the
Zipkin
UI
does
that
fancy
rendering?
So
the
rid
of
that
piece
of
code
in
the
server
is
teeny
tiny,
like
that's,
not
the
complicated
bit
at
the
server
at
all,
but
yeah
the
individual
thing
you're
getting
from
the
client
like
this
spans.
The
server
is
doing
the
join,
which
makes
it
sound
much
grander
than
it.
Videos.
A
Yeah,
yes,
so
you'll
get
partial
traces
all
the
time,
and
you
know
one
one
year,
you'll
come
along
and
you'll
you'll
do
a
pub
and
you'll
get
some
spans
for
things
that
you
haven't
seen
the
root
span
for
you
and
you'll
get
spam
for
traces
which
don't
you
know
which
are
partially
missing
and
so
on.
So
the
model
here
is:
we've
got
immutable
blocks
of
mutable
blocks
and
it's
far
too
complicated.
It
doesn't
need
to
be
that
complicated,
but
yeah.
A
You
definitely
get
partial
data
back
from
the
individual
jobs
and
you
have
to
do
to
join
and
you
have
to
deal
with
the
fact
that
it's
partial.
Thank
you
another
question
from
this
side.
Anyone
ok
open
up
for
the
whole
room.
Ok,
I,
think
we're
done.
I
do
have
some
things
I'm
supposed
to
promote
for
people.
There
is
an
open
tracing
salon.
Please
go
to
it.
It
can
be
awesome
tomorrow
at
2:00
and
we're
hiring
as
usual.
We've
got
offices
in
Berlin,
though,
which
is
cool.